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PREFACE 


This  final  report  describes  activities  performed  in  support  of  the  Air  Force  Research  Laboratory 
System  Control  Interfaces  Branch  (AFRL/HECI)  under  workunit  #71840903.  The  work  was 
accomplished  during  the  period  of  October,  1 999  through  August,  2006  although  the  workunit 
was  zero-funded  in  the  years  FY2004-2006.  This  report  officially  completes  the  workunit  and 
discusses  the  general  program  background  and  significant  technical  accomplishments.  Due  to  the 
loss  of  funding  in  FY2004,  there  was  no  additional  technical  work  accomplished  after  that  time. 
Thus,  the  bulk  of  the  material  contained  herein  has  been  extracted  from  interim  reports  published 
previously. 

An  FDI  research  group  (consisting  of  AFRL/HECI,  General  Dynamics  Advanced  Information 
Engineering  Services,  and  L3  Communications)  provided  human  factors  and  software  experts  for 
the  project.  Their  responsibilities  are  listed  below. 

US  Air  Force  Research  Laboratory  (AFRL/HECI,  WPAFB  OH): 


Antonio  Ayala 
Guy  French,  Ph.D. 

John  Reising,  Ph.D. 

Michael  Snow,  Ph.D. 


Computer  Engineer:  Simulation  Software 

Engineering  Psychologist:  Experimental  Design  Data  Collection, 

Data  Analysis 

Senior  Crew  Systems  Research  Engineer:  System/Display  Format 
Design 

Crew  Systems  Research  Engineer:  System/Display  Format  Design 


General  Dynamics  Advanced  Information  Engineering  Services  (Dayton  OH): 


Noel  Cox 
Joe  Ferraro 
David  Murphy 
Tony  On- 
Mark  Squire 


Senior  Systems  Engineer:  Software  Design,  Data  Collection 
Senior  Systems  Engineer:  Software  Design 
Human  Factors  Engineer:  Data  Collection,  Training 
Senior  Operations  Analyst:  System  Design,  Subject  Matter  Expert 
Systems  Engineer:  Simulation  Software,  Data  Collection 


L3  Communications  (Dayton  OH): 


Sam  Longo 
Thomas  Whitman 


Senior  Simulation  Systems  Engineer:  Hardware  Design 
Senior  Software  Engineer:  Simulation  Software 
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INTRODUCTION 


Data  from  the  Air  Force  Safety  Center  show  that,  from  1990  through  2000,  the  Air  Force  lost  131 
personnel  and  $1.44  billion  in  63  mishaps  falling  into  the  category  of  Controlled  Flight  Into 
Terrain  (CFIT).  A  CFIT  mishap  is  one  in  which  aircrew  fly  a  serviceable  aircraft  into  terrain  due 
to  inadequate  prior  awareness  of  impending  disaster.  Commercial  aviation  has  a  similar  problem, 
suffering  approximately  four  CFIT  mishaps  per  year  in  Western  transports  from  1990  to  2000 
(Phillips,  2001)  with  accompanying  financial  and  human  costs.  Historically,  technological 
approaches  to  preventing  CFIT  mishaps  have  focused  on  warning  and  automatic  avoidance 
systems  (e.g.,  the  Ground  Proximity  Warning  System).  Such  systems  have  been  somewhat 
successful  in  treating  symptoms  of  the  CFIT  disease  -  the  imminent  impact  with  terrain  brought 
about  by  aircrew  loss  of  situation  awareness.  However,  their  lack  of  complete  success  in 
preventing  these  mishaps  may  be  at  least  partially  attributed  to  the  fact  that  they  have  not 
eliminated  CFIT’s  root  cause:  that  aircrew  lose  fundamental  awareness  of  the  relationship 
between  the  aircraft’s  flight  path  and  surrounding  terrain  in  the  first  place.  The  advent  of  newer 
CFIT-prevention  systems  (such  as  the  Enhanced  Ground  Proximity  Warning  System)  will 
improve  terrain  awareness,  but  will  still  require  pilots  to  divert  attention  away  from  primary 
flight  instruments  and  the  out-the-window  scene  and  still  rely  at  least  partially  on  alerts  and 
warnings  to  avoid  CFIT. 

Several  research  and  development  efforts  are  now  being  conducted  worldwide  by  institutions, 
civil  and  military,  to  prevent  this  critical  loss  of  situation  awareness  by  incorporating  a  concept 
known  as  synthetic  vision  in  the  cockpit.  Synthetic  vision  refers  to  the  portrayal  of  information  in 
a  database,  usually  in  perspective  view  and  usually  as  part  of  a  primary  flight  display.  This 
information  can  be  based  on  an  on-board  database  (e.g.,  digital  terrain  elevation  data,  airport  and 
runway  data)  and  can  also  include  elements  of  off-board  data  (e.g.,  datalinked  traffic  information 
or  assigned  approach  path).  The  goal  of  a  synthetic  vision  display  is  to  increase  aircrew  situation 
awareness  by  allowing  a  view  of  surrounding  terrain,  commanded  flight  path,  air  traffic, 
obstacles,  or  any  other  task-relevant  information  that  is  available  and  can  be  incorporated  into 
the  underlying  database.  Fundamentally,  it  is  believed  that  if  pilots  can  accurately  perceive  the 
surrounding  terrain  via  a  synthetic  vision  display,  it  will  greatly  reduce  the  likelihood  that  they 
will  fly  into  it  -  or  get  into  a  situation  that  requires  an  alert  or  warning  to  avoid  doing  so.  Further, 
the  inclusion  of  commanded  flight  paths  in  these  displays  is  designed  to  enable  pilots  to  fly  more 
complex  paths  with  tighter  tolerances  under  IFR  (Instrument  Flight  Rules),  allowing  greater 
airport  throughput  and  better  avoidance  of  noise  abatement  areas. 

Because  synthetic  vision  displays  are  only  as  good  as  the  databases  and  information  that  serve  as 
their  foundation,  military  concepts  for  this  technology  call  for  the  integration  of  synthetic  vision 
with  sensor  information  and  images.  The  two  technologies  are  complementary  in  many  ways. 
Whether  or  not  integration  of  sensor  images  in  civilian  synthetic  vision  displays  will  prove 
practical  and  cost  effective  remains  to  be  seen.  There  are  several  key  technologies  that  enable 
synthetic  vision  displays,  all  maturing  in  the  next  few  years. 

This  report  begins  with  a  discussion  of  the  analysis  of  the  human  and  display  factors  relevant  to 
synthetic  vision  in  a  military  cockpit  environment.  It  then  describes  the  results  of  studies 
examining  the  impact  of  synthetic  vision  displays  on  pilot  situation  awareness  and  workload. 
Concluding  remarks  on  additional  factors  affecting  the  use  of  synthetic  vision  and  a  future 
concept  for  its  implementation  are  described. 
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I.  HUAAAN  FACTORS  IN  SYNTHETIC  VISION  DISPLAYS 


Enabling  technologies  needed  to  provide  pilots  a  synthetic  view  of  the  outside  world  are 
expected  to  mature  within  the  next  three  to  five  years.  Such  a  synthetic  view,  a  display  based  at 
least  partially  on  a  database,  is  expected  to  greatly  increase  aircrew  situation  awareness  in 
reduced  visibility  conditions  with  a  consequent  improvement  in  air  safety.  In  particular,  it  is 
hoped  that  providing  a  synthetic  view  of  the  commanded  flight  path  and  the  terrain  surrounding 
the  flight  path  will  greatly  reduce  the  incidence  of  controlled  flight  into  terrain  accidents.  If  a 
synthetic  view  can  be  provided  to  the  pilot  as  part  of  his/her  head-up  primary  flight  display,  the 
potential  exists  to  provide  these  benefits  while  allowing  the  pilot’s  head,  eyes,  and  attention  to 
remain  focused  outside  the  cockpit. 

This  section  reviews  the  current  status  of  enabling  technologies  for  head-up  synthetic  vision 
displays.  Human  factors  issues  associated  with  the  design  and  display  of  information  are 
discussed  in  detail.  Among  these  issues  are  pilot  information  requirements,  depth  and  self-motion 
cues  provided,  visual  clutter  in  the  head-up  display  (HUD),  format  of  the  synthetic  view 
underlying  other  HUD  symbology,  integration  with  head-down  displays,  integration  of  the 
synthetic  view  with  sensor  information,  and  attentional  issues.  Examples  of  how  these  issues  are 
considered  in  the  design  of  a  synthetic  vision  HUD  are  given,  and  recommendations  made  for  the 
design  of  these  displays  based  on  this  discussion  and  findings  from  human  factors  studies 
conducted  by  AFRL. 

ENABLING  TECHNOLOGIES 
Precision  Navigation  Systems 

In  order  to  utilize  any  navigation  system,  knowing  one’s  current  position  is  highly  desirable.  To 
make  the  best  use  of  an  on-board  terrain  database  for  navigation,  especially  if  trying  to  land  in 
Instrument  Meteorological  Conditions  (IMC),  one  needs  to  know  one’s  position  with  great 
accuracy.  To  achieve  acceptance  from  the  aviation  community,  terrain  databases  need  to  be 
supported  by  a  highly  accurate  system  for  precision  navigation.  Prior  to  the  Global  Positioning 
System  (GPS),  the  ability  to  know  one’s  own  location  was  constrained  by  the  available 
technology,  based  mostly  on  various  radio  beacons.  Aviation  regulations  governing  terminal 
operations  and  instrument  flight  were  drafted  accordingly.  However,  GPS  promises  to  provide 
much  greater  accuracy  and,  as  a  result,  current  regulations  will  become  needlessly  conservative. 

The  National  Air  Space  (NAS)  system  is  receiving  a  major  overhaul  in  order  to  incorporate  GPS 
technologies.  This  overhaul  will  allow  more  precise  navigation  for  all  aircraft  equipped  with 
GPS.  In  simple  terms,  the  NAS  will  rely  on  two  different,  but  related,  systems  for  accurate 
positioning  of  air  vehicles:  WAAS  and  LAAS.  The  Wide  Area  Augmentation  System  (WAAS) 
will  consist  of  roughly  25  ground  reference  stations  that  will  assess  basic  GPS  data  for  errors,  a 
ground  master  station  that  will  calculate  corrections  for  any  detected  errors,  and  a  set  of  satellites 
that  will  broadcast  the  corrections  to  GPS  equipped  air  vehicles.  WAAS  is  expected  to  improve 
basic  GPS  accuracy  from  100  meters  to  about  7  meters.  The  Local  Area  Augmentation  System 
(LAAS)  has  a  similar  configuration  to  that  of  the  WAAS  in  that  it  broadcasts  a  correction  to  the 
GPS  data,  further  improving  accuracy  to  approximately  1  meter  in  equipped  areas.  WAAS  is 
expected  to  provide  accurate  data  for  enroute  navigation,  non-precision  approaches,  and  ILS 
Category  I  approaches  at  some  locations  (http://gps.faa.gov/Programs/WAAS/waas.htm),  while 
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LAAS  will  eventually  provide  the  accuracy,  availability,  and  system  integrity  required  for  ILS 
Category  III  approaches  (http://gps.faa.gov/Programs/LAAS/laas.htm). 

Being  developed  in  parallel  is  a  military  system  compatible  with  WAAS  and  LAAS:  the  Joint 
Precision  Landing  System  (JPALS).  JPALS  is  designed  to  supplement  the  WAAS  and  LAAS 
components  for  military  air  vehicles  when  operating  in  areas  where  there  is  no  WAAS/LAAS 
infrastructure  (e.g.,  OCONUS,  austere  landing  fields,  and  at  sea).  Expected  accuracy  is 
approximately  two  meters,  allowing  approaches  down  to  ILS  Category  II.  JPALS  will  also 
incorporate  anti-jamming  features  and  rapid  set-up  time  for  the  tactical  version.  Additionally,  a 
U.S.  Navy  version  of  JPALS  known  as  Shipboard  Relative  GPS  (SRGPS)  will  increase  accuracy 
to  less  than  1  meter  and  allow  automatic  carrier  landings  (McCarthy  &  Colby,  2000). 

Avionics 

Another  technology  necessary  for  synthetic  vision  to  be  useful  is  a  display  capable  of  showing 
the  terrain  image.  Current  CRT  and  LCD  technologies  are  certainly  capable  of  displaying 
synthetic  imagery  head-down.  However,  it  is  expected  that  incorporating  the  imagery  in  a 
conformal  primary  flight  display  (so  that  elements  in  the  synthetic  image  directly  overlay 
elements  in  the  real  world)  will  yield  the  greatest  benefit  (Wickens  &  Long,  1995).  To  show 
conformal  synthetic  terrain  will  require  some  form  of  head-up  display  (HUD),  either  a 
conventional  HUD  or  a  head-mounted  display  (HMD).  Both  of  these  technologies  have 
limitations.  Current  conventional  HUDs  tend  to  have  narrow  fields  of  view,  limiting  the  amount 
of  the  synthetic  scene  viewable  by  the  pilot  at  any  given  moment.  They  are  also  limited  in  field 
of  regard,  allowing  the  scene  to  be  viewable  only  from  the  HUD  design  eyepoint  (or  eyebox)  on 
aircraft  boresight.  HMDs  have  the  potential  for  much  greater  fields  of  view,  allowing  more  of  the 
scene  to  be  viewed,  and  fields  of  regard  limited  only  by  the  biomechanics  of  the  human  head  and 
neck,  allowing  the  scene  to  be  displayed  off-boresight  as  well.  Current  limiting  factors  of  HMDs 
are  optics  weight  and  bulk,  and  head- tracking:  the  displayed  image  must  be  able  to  keep  up  with 
the  motion  of  the  head,  especially  if  the  real  world  is  even  partly  visible.  Misregistration  in 
overlaid  symbology,  already  an  issue  in  a  HUD  fixed  to  the  aircraft,  becomes  even  more  of  an 
issue  for  an  HMD  that  is  not  fixed  in  space,  but  is  free  to  move  -  not  only  with  intentional  pilot 
head  movements,  but  also  as  a  result  of  helmet  slippage  and  compression  during  high-G 
maneuvers.  Display  factors  will  be  discussed  in  more  detail  in  the  next  Section. 

Databases 

What  shows  up  in  the  head-up  display  can  be  legible  and  updated  as  rapidly  as  the  head  can  be 
moved,  but  if  the  underlying  terrain  database  is  not  accurate,  the  imagery  displayed  still  won’t  be 
entirely  conformal  to  the  real  world,  leading  to  potentially  dangerous  mismatches.  The  terrain  in 
the  database  must  be  accurate  and  of  sufficient  resolution  to  allow  all  terrain  flight  hazards  to  be 
displayed  in  real-time.  The  Shuttle  Radar  Topography  Mission  collected  data  in  an  effort  to 
improve  the  accuracy,  resolution,  and  coverage  of  the  terrain  database  for  NIMA  and  the  civil 
aviation  community.  The  updated  terrain  database  provides  a  resolution  of  approximately  30 
meters  with  <  16  m  absolute  vertical  height  accuracy,  <  10  m  relative  vertical  height  accuracy 
and  <  20  m  absolute  horizontal  circular  accuracy,  between  60N  and  56S 

(http://www.jpl.nasa.gov/srtm/statistics.html).  Of  even  more  concern  than  accuracy  of  terrain 
elevations  is  the  accuracy  of  other  elements  included  in  the  synthetic  vision  display,  especially 
obstacles.  Maintaining  an  accurate  database  that  includes  manmade  changes  in  the  aviation 
environment  (e.g.,  towers,  runway  alterations,  changes  in  controlled  and  restricted  airspace)  is  a 
key  challenge  facing  implementation  of  synthetic  vision  systems. 
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Symbology 

The  last  key  piece  of  a  head-up  synthetic  vision  system  is  the  symbology  displayed  to  the  pilot. 
Regardless  of  other  enabling  technologies  in  the  cockpit,  what  pilots  actually  see  in  head-up 
synthetic  vision  displays  will  be  the  final  determinant  of  how  useful  these  displays  are,  the 
situation  awareness  they  afford  to  pilots,  the  workload  they  impose,  and  -  ultimately  -  how 
safely  aircraft  with  these  systems  will  be  piloted.  This  symbology  has  been  a  primary  focus  of 
work  conducted  in  our  lab  and  others  in  recent  years. 

HUAAAN  FACTORS  ISSUES 

There  is  a  wealth  of  human  factors  issues  associated  with  the  use  of  synthetic  vision  in  cockpits, 
whether  one  is  designing  for  a  head-down  display  or  a  head-up  display.  However,  the  fact  that 
HUDs  are  monochrome  and  meant  to  be  seen  through  makes  them  an  especially  challenging 
display  for  which  to  design  synthetic  vision  symbology.  Three  general  considerations  that  human 
factors  engineers  must  take  into  account  when  designing  these  displays  are,  1)  the  information 
requirements  of  the  pilot,  2)  depth  and  self-motion  cues  provided  by  the  symbology,  and  3) 
display  principles  appropriate  to  the  hardware  available. 

Information  Requirements 

Most  synthetic  vision  displays  currently  under  research  and  development  include  more  than  just 
a  view  of  synthetic  terrain.  To  minimize  attention  management  requirements  (and  associated 
workload),  designers  usually  combine  synthetic  vision  with  a  display  to  which  the  pilot  might 
otherwise  have  to  shift  attention.  One  common  example  of  this  is  including  primary  flight 
information  in  a  synthetic  vision  display.  This  primary  flight  information  often  incorporates  a 
perspective  view  of  the  commanded  flight  path;  the  technologies  enabling  synthetic  vision  also 
enable  such  tunnel-  and  highway-in-the-sky  displays.  Other  examples  of  information  included  in 
synthetic  vision  displays  include  traffic,  weather,  and  restricted  airspace:  currently  all  shown  on 
cockpit  displays  that  require  division  of  attention  between  the  primary  flight  display  and  the  view 
out  the  window. 

Information  required  by  the  pilot,  and  therefore  considered  for  inclusion  in  a  synthetic  vision 
display,  depends  on  the  pilot’s  task  at  hand.  A  pathway-in-the-sky  that  might  be  useful  when 
flying  a  complex,  precision  LAAS  approach  could  also  be  one  of  the  first  things  a  pilot  might 
declutter  from  the  HUD  when  cruising  at  altitude.  The  information  requirements  also  depend 
greatly  on  the  role  and  mission  of  the  aircraft. 

One  key  difference  in  these  roles  and  missions  is  whether  the  aircraft  and  crew  are  civilian  or 
military.  With  regard  to  primary  flight  information,  the  requirements  and  standards  for  civil 
aviation  (FAR  25.1303,  Federal  Aviation  Administration,  2000)  and  military  aviation  (MIL-STD 
1787C,  U.S.  Department  of  Defense,  2000)  are  essentially  the  same.  However,  sensors  are  more 
prevalent  in  the  military  aviation  community  than  in  the  civil  aviation  community.  A  display  of 
synthetic  imagery  designed  for  military  use  must  be  able  to  incorporate  sensor  imagery  and,  as 
sensor  technologies  become  more  cost  effective,  synthetic  vision  formats  for  civil  aviation 
should  also  have  this  capability.  The  integration  of  imagery  based  on  sensors  with  imagery  based 
on  a  database  has  implications  for  the  human  factors  engineering  of  these  displays,  driven  by  the 
relative  strengths  and  weakness  of  each  technology.  Therefore,  a  synthetic  vision  display 
incorporating  sensor  imagery  should  allow  the  pilot  to: 
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1.  Display  sensor  imagery,  synthetic  imagery,  both  combined,  or  neither  at  his/her 
discretion. 

2.  Immediately  and  clearly  discern  the  source  of  information  in  the  image  (sensor  or 
database). 

A  synthetic  image  is  only  as  good  as  the  database  and  navigation  system  that  serves  as  its  source; 
similarly,  a  sensor  image  is  only  as  good  as  the  sensor  and  the  environmental  conditions  in  which 
it  is  operating.  Meteorological  conditions,  phase  of  flight,  and  type  of  terrain  can  all  affect  the 
relative  usefulness  of  each  image  and  image  source.  The  format  of  the  synthetic  vision  display 
and  the  interface  used  to  control  it  should  support  pilot  decision-making  with  regard  to  which 
image,  if  any,  to  display. 

To  successfully  meet  the  needs  of  aircrew,  the  synthetic  vision  display  should  not  simply  display 
data:  it  should  display  information.  One  example  of  the  difference  between  these  two  is  the 
format  chosen  for  synthetic  terrain.  Rather  than  simply  portraying  a  terrain  database  (even  one 
with  a  photorealistic  overlay),  the  synthetic  vision  display  should  attempt  to  portray  what  the 
aircrew  needs  to  know  about  that  data:  the  height  of  terrain  relative  to  the  aircraft,  the  location  of 
obstacles  that  may  be  of  concern,  and  the  relationship  between  surrounding  terrain  and  the 
aircraft’s  commanded  flight  path. 

One  key  concern  related  to  pilot  information  requirements  is  the  possibility  of  attentional 
tunneling  or  cognitive  capture:  the  idea  that  a  synthetic  vision  display,  especially  one  including  a 
pathway  or  tunnel,  will  be  so  compelling  and  contain  so  much  of  the  pilot’s  information 
requirements  that  awareness  of  other  displays  and  events  will  deteriorate  in  comparison.  There  is 
much  research  still  to  be  done  in  this  area.  Some  authors  report  no  difference  in  situation 
awareness  on-path  versus  off-path  (Snow  &  Reising,  1999),  while  others  report  a  decrease  in 
awareness  of  events  and  information  not  contained  in  the  synthetic  vision  display  (Olmos,  Liang, 
&  Wickens,  1997;  Williams,  2000).  To  date,  most  of  these  studies  have  been  done  with  HDDs 
and  the  differences  found  (or  not  found)  seem  to  be  highly  task-dependent.  It  remains  to  be  seen 
whether  using  HUDs  for  synthetic  vision,  and  presumably  focusing  pilots’  attention  head-up 
rather  than  head-down  will  alleviate  these  concerns. 

Depth  And  Self-Motion  Cues 

One  concept  of  synthetic  vision  is  that  texture-mapping,  satellite  imagery,  and  terrain  databases 
should  be  used  to  create  a  display  that  mimics  daytime  flight  in  Visual  Meteorological 
Conditions  (VMC).  While  such  a  display  is  certainly  useful,  there  have  been  instances  of  CFIT  in 
VMC,  and  even  daytime  VMC.  Pilots  flying  over  open  water,  desert,  or  snow  often  lack  the 
normally  rich  visual  cues  that  humans  use  to  accurately  perceive  depth  and  self-motion. 
Knowledge  of  these  cues  allows  human  factors  engineers  to  create  synthetic  vision  display 
formats  that,  in  many  cases,  allow  pilots  to  better  perceive  depth  and  self-motion  than  if  they 
were  looking  out  the  window  in  daytime  VMC  conditions.  A  brief  description  of  some  of  the 
depth  and  self-motion  cues  that  may  be  usefully  incorporated  into  a  head-up  synthetic  vision 
display  follows. 

Perspective  or  splay  -  Perspective  refers  to  the  angle  of  lines  parallel  to  the  observer’s  viewpoint. 
A  common  example  of  this  used  in  introductory  perception  texts  is  the  observer  standing  in  the 
middle  of  a  railroad  track.  The  closer  the  observer  is  to  the  ground  the  greater  will  be  the  angle 
formed  by  the  two  rails.  Perspective  has  been  shown  to  be  a  critical  cue  in  perception  of  altitude 
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by  pilots  (Flach,  Warren,  Gamess,  Kelly,  &  Stanard,  1997).  Indeed,  student  pilots  are  sometimes 
told  upon  landing  to  look  at  a  distant  point  down  the  runway  to  gauge  height  above  it. 

Depression  angle  -  Depression  angle  is  the  perceived  distance  between  lines  perpendicular  to  the 
observer’s  viewpoint.  Lines  closer  to  the  horizon  are  closer  together. 

Ground  intercepts  -  Several  authors  have  proposed  displaying  ground  intercepts  or  drop  lines 
between  airborne  objects  and  their  two-dimensional  location  on  the  ground,  both  in  static  and 
dynamic  displays.  This  seems  especially  helpful  in  judging  relative  distances  or  depth  (Hendrix 
&  Barfield,  1995). 

Apparent  size  -  When  objects  of  a  known  or  expected  size  (e.g.,  aircraft,  path  blocks)  are 
presented,  larger  objects  will  be  perceived  as  closer.  The  object  must  be  something  with  which 
the  observer  is  familiar,  especially  in  terms  of  its  size.  As  an  example,  the  perceived  distance  to  a 
runway  shown  in  a  synthetic  vision  HUD  will  be  based  on  the  pilot’s  familiarity  with  how  big 
runways  appear  at  certain  distances. 

Object  interposition  -  In  the  natural  world,  it  is  a  perceptual  constant  that  an  object  closer  to  an 
observer  will  appear  in  front  of  any  object  behind  it. 

Motion  parallax  -  Another  perceptual  constant  in  the  natural  world  is  that  objects  closer  to  an 
observer  will  show  greater  apparent  motion  relative  to  objects  farther  away.  A  relevant  example 
of  a  display  in  which  this  is  a  particularly  strong  cue  is  that  of  a  pathway-in-the-sky 
superimposed  on  a  synthetic  terrain  grid.  Lines  perpendicular  to  the  pilot’s  viewpoint  in  the 
pathway  move  more  quickly  in  the  pilot’s  field  of  view  than  corresponding  lines  in  the 
underlying  synthetic  terrain  grid. 

Optical  flow  -  Optical  flow  refers  to  the  pattern  and  direction  of  movement  of  elements  in  the 
visual  field.  Optical  flow  slowly  expanding  from  the  center  of  the  observer’s  field  of  view 
indicates  slow  movement  directly  ahead.  Rapid  optical  flow  from  left  to  right  with  no  expansion 
would  imply  that  the  observer  is  stationary,  but  pivoting  to  the  left.  Optical  flow  rate  has  been 
found  to  be  a  key  determinant  of  perceived  self-motion  or  egospeed  (Dyre,  1997). 

Discontinuity  rate  -  Discontinuity  rate  refers  to  the  rate  at  which  edges  (or  texture  elements)  in 
the  visual  field  pass  a  fixed  reference  point  (e.g.,  the  boundary  of  a  HUD  field  of  view).  Given  a 
texture  or  grid  of  fixed  size,  discontinuity  rate  will  be  directly  related  to  how  fast  the  observer  is 
moving  and  can  be  key  to  accurate  perception  of  egospeed  (Ballard,  Roach,  &  Dyre,  1998). 

Relative  height  -  Everything  else  being  equal,  objects  higher  in  the  observer’s  field  of  view 
(closer  to  the  horizon)  appear  farther  away  than  objects  lower  in  the  observer’s  field  of  view. 

Texture  gradient  -  When  a  texture  is  present,  as  in  many  synthetic  vision  portrayals,  texture 
elements  will  appear  larger  at  shorter  distances  and  smaller  at  farther  distances.  The  usefulness  of 
this  cue  depends  greatly  on  the  texture  (especially  the  granularity  or  size  of  texture  elements)  and 
the  distance  from  the  observer  to  the  texture.  Given  enough  distance  (or  altitude),  this  cue  to 
depth  and  distance  becomes  useless  as  texture  elements  blend  into  obscurity  (Surdick,  Davis, 
King,  &  Hodges,  1997). 

Relative  brightness  -  Objects  in  the  observer’s  field  of  view  that  are  brighter  tend  to  appear 
closer.  In  a  monochrome  raster-capable  HUD,  making  nearer  objects  (such  as  a  pathway) 
brighter  than  more  distant  objects  (such  as  synthetic  terrain)  will  reinforce  correct  perception  of 
their  relative  distances. 
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Display  Principles 

There  are  many,  many  display  principles  that  have  emerged  from  human  factors  research  over 
the  years.  Not  all  apply  to  HUDs  or  synthetic  vision  displays;  some  apply  to  these  displays 
uniquely.  Those  that  are  (in  the  authors’  view)  most  relevant  are  outlined  below. 

Minimize  clutter  -  The  primary  reason  that  HUDs  exist  is  to  allow  pilots  to  simultaneously 
attend  to  information  displayed  in  the  HUD  and  events  in  the  external  visual  scene.  To  the  extent 
that  one  allows  HUD  symbology  to  obscure  the  external  visual  scene,  one  might  as  well  put  the 
same  symbology  head-down,  save  the  weight  and  cost  of  a  HUD,  and  take  advantage  of  the  color 
capabilities  of  most  HDDs.  Every  single  symbol  element  in  a  HUD  should  be  subjected  to  the 
question:  “Does  the  pilot  need  to  see  this  if  it  means  blocking  the  view  through  the  window?”  If 
the  answer  is  no,  the  symbol  element  should  probably  be  declutterable  and  consideration  should 
be  given  to  discarding  it  entirely.  Performance  suffers  when  task-irrelevant  clutter  is  displayed  in 
a  HUD  (Ververs  &  Wickens,  1998). 

It  should  be  noted  that  minimizing  clutter  is  not  simply  equivalent  to  reducing  the  number  of 
pixels  in  the  pilot’s  field  of  view  at  any  given  moment.  Consideration  must  be  given  to  the  shape 
and  content  of  what  is  displayed.  The  literature  on  visual  search  and  visual  perception  leads  us  to 
conclude  that  some  symbology  elements  will  be  more  confiisable  than  others  (Wickens,  1992). 
Even  though  there  may  be  relatively  few  symbol  elements  in  the  display,  pilots  will  likely 
describe  it  as  cluttered  or  “busy”  if  elements  are  easily  confused  because  of  their  shape,  size, 
position,  or  movement. 

Conformality  -  One  display  principle  unique  to  HUDs  is  that  of  conformality:  to  the  extent 
possible,  symbol  elements  in  the  HUD  should  be  conformal,  or  analogous,  to  elements  in  the 
external  visual  scene.  Examples  of  conformal  symbology  include  the  artificial  horizon,  flight 
path  marker,  and  runway  outline.  A  growing  body  of  research  indicates  that  the  more  conformal  a 
symbology  set  is,  the  better  pilots  will  be  at  simultaneously  attending  to  the  HUD  symbols  and  to 
events  in  the  external  visual  scene,  especially  if  those  events  are  unexpected  (Foyle,  McCann, 
Stanford,  &  Schwirzke,  1993;  Martin-Emerson  &  Wickens,  1997;  McCann,  Foyle,  &  Johnston, 
1993;  Wickens,  1997). 

Consistency  -  Another  display  principle  that  applies  once  a  HUD  is  introduced  to  the  cockpit  is 
that  of  consistency  between  the  HUD  and  HDD,  especially  in  the  case  of  primary  flight  displays. 
The  format  of  information  that  is  similar  in  these  displays  should  be  consistent,  if  not  identical. 
To  the  extent  possible,  pilots  should  not  be  required  to  translate  between  head-up  and  head-down 
display  formats:  the  transition  should  be  seamless.  Otherwise,  needless  workload  is  imposed  on 
the  pilot  along  with  the  possibility  of  misinterpretation. 

Gestalt  perception  -  Applied  in  the  current  context,  gestalt  perception  means  designing 
symbology  so  that  significant  events  or  trends  are  evident,  not  from  examination  and 
interpretation  of  the  individual  symbol  elements,  but  from  the  pattern  of  these  elements  or  their 
movement.  This  allows  perception  of  the  event  or  trend  via  a  global,  automatic,  preattentive 
process.  As  an  example,  consider  an  airplane  that  has  just  lost  power.  A  pilot  should  be  able  to 
discern  this,  not  through  lengthy  examination  and  interpretation  of  individual  HUD  symbol 
elements,  but  at  a  glance  via  the  distinct  pattern  of  movement  of  these  elements  associated  with  a 
loss  of  power  (e.g.,  airspeed,  altitude,  and  vertical  velocity  all  move  counter-clockwise  while 
climb-dive  marker,  flight-path  marker,  and  acceleration  carat  all  move  downward  in  the  HUD 
field  of  view).  A  related  concept  is  that  of  emergent  features  (Wickens,  1992),  the  idea  that 
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displays  should  be  designed  so  that  significant  trends,  events,  or  states  emerge  from  underlying 
individual  display  elements  to  be  perceived  globally  and  preattentively. 

Symmetry  -  The  concept  of  display  symmetry  is  one  example  of  an  emergent  feature  that  is 
particularly  relevant  in  the  current  context.  Display  elements  may  be  designed  so  that  symmetry 
is  achieved  when  a  desired  aircraft  state  is  achieved  (e.g.,  pitch  and  bank  steering  bars  form  a 
symmetric  cross  when  an  aircraft  is  precisely  on  course  and  on  glide-slope).  An  example  of  this 
in  a  synthetic  vision  display  is  proposed  by  Theunissen  (1997).  Head-down  tunnel-in-the-sky 
displays  may  be  designed  so  that  symmetry  is  achieved  relative  to  the  frame  of  the  display.  This 
allows  the  pilot  to  determine  that  the  aircraft  is  in  straight  and  level  flight  and  on  course  at  a 
glance:  if  it  is,  the  lines  of  the  tunnel  will  be  parallel  to  and  equidistant  from  the  edges  of  the 
display  frame.  If  it’s  not,  this  symmetry  will  be  broken. 

Implications  For  Design 

How  are  all  of  these  issues  synthesized  to  create  a  useful  head-up  synthetic  vision  display?  We 
will  illustrate  by  stepping  through  two  design  decisions  that  have  led  to  formats  tested  at  AFRL. 

Path  format  -  We  start  by  examining  pilot  information  requirements.  Among  the  things  a  pilot 
needs  to  know  about  his/her  commanded  path  are: 

•  Where  it  is  now 

•  Where  it  is  going 

•  Where  the  aircraft  is  in  relation  to  it 

•  Where  the  aircraft  will  be  in  relation  to  it  in  the  future 

To  meet  these  information  requirements,  we  could  simply  draw  a  perspective  view  of  a  line  and 
show  a  couple  of  dots  for  where  the  aircraft  is  now  and  where  it  will  be  in  the  future.  This  would 
certainly  minimize  clutter.  However,  the  commanded  path  is  actually  four  dimensional:  the  pilot 
has  an  assigned  altitude,  course,  heading  and  airspeed.  Therefore,  the  path  format  should  show 
the  pilot  where  these  are  and  where  the  aircraft  is  in  relation  to  each  of  them.  These  requirements 
may  be  met  by  either  the  pathway  format  shown  in  Figure  1  or  the  tunnel  format  shown  in  Figure 
2. 
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Figure  1 .  Pathway-in-the-sky  synthetic  vision  format  for  head-up  display. 


Figure  2.  Tunnel-in-the-sky  synthetic  vision  format  for  head-down  display  (adapted  from 

Theunissen,  1997). 
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While  the  basic  information  content  of  these  displays  is  similar,  it  is  interesting  to  note  the 
disparities  between  them,  especially  in  the  context  of  the  depth  and  self-motion  cues  and  display 
principles  discussed  previously.  With  regard  to  the  commanded  path,  both  displays  contain 
strong  splay,  depression  angle,  motion  parallax,  and  discontinuity  rate  cues.  Both  displays 
include  ground  intercepts,  accurate  object  interposition,  and  relative  height  cues.  By 
incorporating  a  “follow-me”  aircraft  of  known  size,  the  pathway  display  incorporates  the  depth 
cue  of  apparent  size  in  what  is  essentially  an  indicator  of  commanded  airspeed  and  altitude. 
However,  unlike  the  tunnel  display,  it  does  not  show  all  dimensions  of  the  pilot’s  commanded 
corridor  (assigned  altitude  block  is  not  overtly  displayed)  and  it  does  not  have  the  extra 
discontinuity  rate  and  splay  cues  provided  by  the  lines  making  up  the  sides  and  top  of  the  tunnel. 

What  about  consistency?  The  default  answer  would  be  that  if  the  designer  chooses  a  pathway 
head-up,  then  a  pathway  should  be  displayed  head-down.  Similarly,  if  the  designer  chooses  a 
tunnel  head-down,  then  a  tunnel  should  be  displayed  head-up. 

However,  we  see  that  the  tunnel  display  involves  more  clutter  than  the  path  display:  were  the 
tunnel  to  be  displayed  monochrome  and  head-up,  its  elements  would  be  less  easy  to  separate 
perceptually  and  the  pilot  would  have  to  peer  through  a  great  deal  more  green  to  see  the  view 
through  the  window.  On  the  other  hand,  whether  viewed  against  an  HDD  frame  or  a  HUD 
combiner  glass  frame,  the  tunnel  does  provide  better  emergent  features  in  the  form  of  symmetry 
and  gestalt  perception. 

Is  there  a  way  to  realize  the  advantages  of  a  tunnel  without  the  associated  clutter  -  and  achieve 
consistency  between  head-up  and  head-down  displays?  If  one  considers  that  “consistency”  does 
not  mean  “identity”,  the  answer  is  yes.  One  way  of  achieving  this  would  be  to  display  a  less 
cluttered  version  of  the  tunnel  head-up  and  superimpose  this  symbology  on  the  head-down 
display.  This  allows  the  pilot  to  transition  between  head-up  and  head-down  displays  without  also 
having  to  make  a  mental  transition  between  display  formats.  This  is  especially  important  in  the 
context  of  a  synthetic  vision  display  because  the  geometric  field  of  view  (i.e.,  the  amount  of 
database  displayed)  will  likely  be  two  to  three  times  wider  in  the  HDD  than  in  the  HUD  (where  it 
must  be  the  same  as  the  HUD’s  physical  field  of  view  to  remain  conformal)  and  pilots  will 
therefore  have  an  even  greater  need  for  consistent  display  symbology  landmarks.  A  notional 
example  is  shown  in  Figure  3.  In  this  case,  the  head-up  display  of  the  commanded  path  contains 
only  the  symbology  elements  shown  in  bright  green  (i.e.,  “HUD  green”)  on  the  head-down 
display. 
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Figure  3.  Notional  example  of  how  to  achieve  consistency  between  head-up  and  head-down 

commanded  path  displays. 


Synthetic  terrain  format  -  Showing  pilots  the  terrain  surrounding  the  flight  path  seems  like  a 
good  way  to  increase  their  terrain  awareness  and  prevent  CFIT,  but  what  is  the  best  way  to  do 
this?  Two  common  approaches  (illustrated  in  Figures  1  and  2)  are  to  superimpose  either  a  texture 
(sometimes  containing  photorealistic  satellite  imagery)  or  a  grid  (illustrated  in  Figure  4)  on  an 
on-board  digital  terrain  elevation  database. 

Again,  let’s  examine  these  two  approaches  in  the  context  of  the  previous  discussion  of  human 
factors  issues.  With  regard  to  information  requirements,  a  photorealistic  texture  seems  better 
suited  to  the  display  of  landmarks  -  if  there  are  landmarks  to  be  displayed  (a  photorealistic 
texture  may  not  be  as  useful  during  low-level  flight  over  water,  desert,  or  other  featureless 
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terrain).  For  other  pilot  information  requirements,  especially  the  possible  need  to  simultaneously 
view  sensor  images  and  differentiate  between  sensor  imagery  and  database  imagery,  a 
photorealistic  texture  may  actually  be  counterproductive:  it  could  obscure  the  sensor  image  or 
elements  of  the  photorealistic  texture  could  be  confused  with  elements  of  the  sensor  image.  A 
grid,  on  the  other  hand,  may  be  superimposed  on  a  sensor  image  without  completely  obscuring  it. 
A  grid  also  allows  the  difference  between  sensor  imagery  and  database  imagery  to  remain  readily 
apparent. 

With  regard  to  depth  and  self-motion,  the  only  two  cues  that  a  photorealistic  texture  would 
provide  that  a  grid  might  not  are  apparent  size  (assuming  visible  landmarks  of  known  size)  and 
texture  gradient.  It  provides  much  weaker  cues  to  perspective,  depression  angle,  motion  parallax, 
and  discontinuity  rate  than  a  grid  format. 

Perhaps  the  greatest  strength  of  a  grid  format  relative  to  a  texture  format  is  clutter  minimization. 
To  the  extent  that  there  is  a  view  to  be  seen  out  the  window,  a  texture  format  will  obstruct  it 
much  more  than  a  grid  format  will. 

Given  a  decision  to  portray  synthetic  terrain  as  a  grid,  what  should  the  grid  look  like?  How  large 
should  the  grid  squares  be?  The  default,  system-centered  answer  would  be  to  simply  “connect  the 
dots”  in  the  underlying  digital  terrain  elevation  database.  If  the  database  contains  elevation 
postings  every  100  meters,  then  the  grid  would  contain  lines  every  100  meters.  This  approach  is 
shown  in  Figure  4. 


Figure  4.  Grid  format  synthetic  terrain  with  lines  every  100  meters. 


A  better,  user-centered  approach  is  to  take  into  account  the  human  factors  issues  previously 
described.  Three  primary  considerations  in  this  case  are:  1)  Pilots’  information  requirement  of 
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accurately  gauging  their  altitude  (or  distance)  relative  to  the  displayed  terrain,  2)  Minimizing 
clutter,  and,  3)  Maximizing  depth  and  self-motion  cues.  The  second  consideration,  minimizing 
clutter,  dictates  that  as  few  lines  as  possible  constitute  the  grid.  The  less  clutter  associated  with 
the  grid,  the  better  the  pilot  will  be  able  to  see  either  an  underlying  sensor  image  or  the  view 
through  the  window.  The  second  consideration,  based  on  existing  literature  (Dyre,  1997;  Flach  & 
Warren,  1993;  Flach  et  ah,  1997;  Foyle  &  Kaiser,  1991;  Foyle,  Kaiser,  &  Johnson,  1992;  Kelly, 
1993;  Kelly,  Flach,  Gamess,  &  Warren,  1993),  dictates  that: 

•  Splay  being  a  very  important  cue  to  perceived  altitude,  at  least  two  longitudinal  lines 
(lines  parallel  to  the  observer’s  line  of  sight)  should  always  be  visible  in  the  HUD  field  of 
view  to  accurately  provide  splay  cues.  Functionally,  this  means  that  the  grid  should  be 
sized  so  that  the  center  of  the  HUD  field  of  view  spans  two  full  grid  squares  (or  three 
longitudinal  grid  lines).  This  guarantees  that  at  least  two  longitudinal  grid  lines  will 
always  be  available  in  the  HUD  field  of  view  to  show  splay. 

•  Optical  flow,  while  critical  to  accurate  perception  of  self-motion  (a.k.a.,  egospeed)  and 
changes  in  acceleration,  serves  as  visual  noise  with  respect  to  perception  of  altitude  and  is 
inversely  related  to  accuracy  of  altitude  control.  Functionally,  this  means  that  grid  density 
should  be  minimized  to  reduce  the  number  of  elements  in  the  visual  flow  field. 

•  Edge  discontinuity  and  depression  angle,  while  important  factors  in  determining 
egospeed  and  altitude,  will  be  functions  of  splay  and  optical  flow  rate  (respectively)  in  a 
square  grid  and  thus,  as  completely  redundant  cues,  need  not  be  separately  factored  into 
the  calculation  of  optimum  grid  size. 

•  Because  apparent  size  is  only  an  accurate  cue  when  objects  (in  this  case  grid  squares) 
have  some  known  or  expected  size,  grid  dimensions  should  remain  constant  and  not 
change  as  a  function  of  altitude. 

To  calculate  the  grid  size  needed  to  guarantee  two  full  squares  in  the  HUD  field  of  view,  one 
must  assume  a  reference  altitude  AGE,  a  lateral  HUD  field  of  view,  and  a  maximum  downward 
viewing  angle  over  the  aircraft  nose  and  through  the  HUD.  Assume  an  altitude  AGE  of  1500  feet 
(a  standard  pattern  altitude  for  fighter/attack  aircraft  and  other  military  jet  aircraft),  a  lateral 
HUD  field  of  view  of  30  degrees  (the  field  of  view  of  the  LANTIRN  HUD  aboard  F-16s 
equipped  with  the  LANTIRN  system),  and  a  maximum  downward  viewing  angle  of  14  degrees 
at  zero  pitch  (based  on  the  14-20  degree  downward  viewing  angle  actually  found  in  most  current 
fighter/attack  aircraft). 

Let  a  equal  the  altitude  AGL,  /  equal  the  lateral  field  of  view  of  the  HUD,  and  v  equal  the 
maximum  downward  viewing  angle  over  the  aircraft  nose  and  through  the  HUD.  We  must  first 
calculate  the  minimum  distance  to  the  ground  viewable  through  the  HUD  {d)\ 

d  =  fl(tan(90  -  v))  =  1500(tan(14))  =  6016  feet 

The  total  width  of  the  ground  viewable  through  the  HUD  (w)  then  becomes: 

w  =  d(tan(/)  =  60 1 6(tan(30))  =  3224  feet 

Dividing  this  in  half  (to  guarantee  two  full  grid  squares  in  the  HUD  field  of  view)  and  converting 
to  meters  yields  a  grid  size  of  roughly  500  meters,  as  shown  in  Figure  5. 
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Figure  5.  Grid  format  synthetic  terrain  with  lines  every  500  meters. 


The  above  grid  size  is  less  eluttered  than  that  in  Figure  4,  but  still  contains  the  same  depth  and 
self-motion  cues.  It  should  be  noted  that  this  example  is  limited  and  does  not  take  into  account 
other  pilot  information  requirements  such  as  the  density  and  size  of  terrain  features  being 
overflown.  Also,  the  grid  size  is  optimized  only  for  a  fighter/attack  aircraft  flying  at  a  standard 
pattern  altitude.  The  parameters  of  the  grid  would  change  as  parameters  of  the  HUD,  aircraft,  and 
reference  altitude  change. 

RECOAAMENDATIONS 

•  Consider  a  grid  format  for  synthetic  terrain.  This  maximizes  depth  and  self-motion  cues, 
minimizes  clutter,  and  allows  simultaneous  view  of  a  sensor  image  without  confusion  as  to 
the  source  of  the  imagery. 

•  Regardless  of  which  pathway  or  tunnel  format  is  chosen  (if  any),  ensure  that  the  only 
transition  required  of  the  pilot  as  he/she  transitions  between  head-up  and  head-down  displays 
is  eye  movement  (i.e.,  ensure  a  seamless  mental  transition).  One  way  to  achieve  consistency 
between  head-up  and  head-down  displays  is  for  the  head-up  display  to  show  a  minimal  set  of 
the  head-down  symbology. 

•  Where  feasible,  symbology  design  should  allow  pilots  to  process  the  display  preattentively 
(i.e.,  using  global  or  gestalt  perception).  Emergent  features  such  as  symmetry  should  be  used 
to  indicate  significant  events  or  aircraft  states. 
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The  advent  of  synthetic  vision  in  the  cockpit  is  imminent.  Synthetic  vision  systems  represent  a 
means  to  solve  the  problem  of  controlled  flight  into  terrain,  a  problem  as  old  as  human  aviation 
and  one  that  continues  to  prove  extremely  costly  in  both  military  and  civil  aviation.  Enabling 
technologies  are  all  maturing  and  several  substantial  efforts  are  now  under  way  to  transition 
synthetic  vision  systems  from  laboratories  to  cockpits.  Care  must  be  taken  as  these  systems  are 
developed  and  fielded  to  take  into  account  human  factors  issues  critical  to  the  usability  and 
safety  of  these  displays. 

In  the  next  Section,  human  factors  issues  related  to  the  display  hardware  are  discussed. 
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II.  DISPLAY  REQUIREMENTS  FOR  SYNTHETIC  VISION 


The  on-going  revolution  in  information  availability  is  not  limited  to  email  and  web  based 
eommunications,  but  is  also  occurring  in  increasingly  crowded  skies.  Aircrews  will  be  expected 
to  effectively  exploit  new  and  updated  information  from  on-  and  off-board  sources  to  improve 
the  likelihood  of  mission  success.  The  capabilities  of  displays  on  board  aircraft  and  the 
information  presentation  format  will  affect  aircrew  ability  to  manipulate  and  manage  this  new 
wealth  of  information.  As  discussed  in  Section  I,  one  set  of  systems  that  will  impact  the  use  of 
displays  in  the  cockpit  in  the  next  decade  falls  under  the  heading  of  Synthetic  Vision  Systems 
(SVS).  Each  SVS  involves  the  use  of  on-board  databases  that  may  contain  detailed  information 
about  flight  hazards  (e.g.,  terrain),  navigational  aids,  tactical  elements,  and  flight  plan.  The 
information  in  these  databases  can  then  be  presented  to  aircrew  to  enhance  weapon  system 
effectiveness  by  providing  more  accurate  and  current  data,  increasing  crew  situation  awareness, 
optimizing  workload,  and  increasing  system  flexibility.  As  more  information  becomes  available 
and  storage  capacity  of  on-board  systems  increases,  the  pressure  to  display  information  in  an 
effective  manner  rises. 

Related  to  this  is  the  movement  to  open  military  aircraft  to  new  off-board  data  sources,  loosely 
given  the  name  “Real  Time  Information  in  the  Cockpit  (RTIC)”.  The  RTIC  concept  assumes  a  set 
of  enabling  technologies:  1)  a  datalink  capability,  2)  an  accurate  geopositioning  capability — 
such  as  the  Global  Positioning  System  (GPS),  and  its  more  accurate  variant.  Differential  GPS 
(DGPS) —  3)  a  robust  and  distributed  Command,  Control,  Communications,  Computers,  & 
Intelligence  (C4I)  system,  and  4)  sufficient  display  space  and  capability.  Specifically,  RTIC 
proposes  to  provide  timely  and  reliable  information  regarding  any  number  of  pertinent  factors, 
including,  but  not  limited  to:  defensive  and  offensive  air-to-air  and  surface-to-air  threats,  hostile 
force  locations,  mission  re-routing,  weather  updates,  information  on  friendly  forces, 
communications,  support,  and  mission  changes.  Initial  use  of  new  information  sources  will  likely 
be  on  head-down  displays  (HDDs),  with  eventual  incorporation  in  Head-Up  and  Head-Mounted 
displays  (HUDs  and  HMDs,  respectively). 

One  issue  raised  by  the  expanded  availability  of  information  is  the  set  of  modalities  used  to 
present  it  to  the  aircrew.  One  option  of  course,  is  to  simply  place  a  new  display  and/or  computer 
into  the  cockpit.  As  an  example,  the  Airborne  Broadcast  Intelligence  (ABI)  system  is  one  such 
RTIC  system  that  fuses  available  near  real-time  intelligence,  digital  mapping,  and  imagery 
information  and  displays  it  in  the  cockpit.  In  its  current  state  of  development,  ABI  is  a  “strap-on” 
computer  workstation  that  utilizes  a  traditional  desktop  display  employing  drop-down  menus  and 
“hot  button”  interfaces.  As  such,  use  of  the  ABI  system  currently  involves  an  additional 
dedicated  crewmember,  responsible  for  ABI  system  operation  and  communication  of  the  relevant 
information  to  the  flight  crew.  Because  real  estate  in  the  cockpit  is  limited,  there  are  only  so 
many  new  displays,  computers,  and  crewmembers  that  can  be  added.  Perhaps  a  better  option  is  to 
integrate  the  on-  and  off-board  data  flow  into  flight  system  displays  and  make  it  available  to  the 
entire  aircrew.  This  option  places  more  demands  on  the  displays  in  the  cockpit,  some  of  which 
are  not  up  to  the  task.  More  capable  displays  will  eventually  be  required  to  handle  the  new  level 
of  information  that  aircrews  will  have  and  need  to  perform  their  missions.  The  purpose  of  this 
section  is  to  describe  some  of  the  information  and  display  requirements  associated  with  new 
technologies  expected  to  reach  military  cockpits  in  the  next  decade. 
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HEAD-DOWN  DISPLAY  REQUIREMENTS 


With  regard  to  HDDs,  the  physical  characteristics  required  for  a  synthetic  vision  display  are  not 
significantly  different  from  those  that  have  become  standard  in  current  glass  cockpits. 
Performance  requirements  driving  cockpit  display  technology  include  sunlight-readability  and/or 
night  vision  compatibility — neither  of  which  is  present  in  consumer  displays.  Hopper  has 
examined  performance  requirements  for  defense  applications  and  compared  aircraft  cockpits  to 
other  crewstations.  For  research  purposes,  a  commercial  off-the-shelf  (COTS)  color  active- 
matrix  liquid  crystal  display  (AMLCD),  cathode  ray  tube  (CRT),  or  other  display  technology 
defines  a  baseline  for  visual  display  performance.  Certified  aircraft  cockpit  display  hardware, 
however,  is  custom  designed  and  manufactured,  albeit  commercial  fabrication  facilities  and 
components  are  used  wherever  possible.  Night  vision  filters  for  backlights,  dual  backlighting  for 
day/night  mode,  anti-reflective  coatings,  electromagnetic  interference  filters,  and  dimming 
electronics  are  all  necessary  in  military  cockpit  displays  but  are  simply  not  included  in  COTS 
displays.  These  additional  features  degrade  the  visual  performance  of  the  underlying  COTS 
display.  This  degradation  must  be  factored  into  efforts  to  correlate  laboratory  research  to  flyable 
hardware.  Efforts  to  integrate  display  components  designed  for  consumer  products  are  always 
underway  but  typically  fail  to  achieve  certification — or  worse,  deliver  performance 
disappointment  to  aircrews  (degrading  combat  performance  and  engendering  new  development 
efforts). 

The  resolution  and  color  capabilities  of  head-down  synthetic  vision  display  formats  may,  or  may 
not,  be  especially  demanding.  Some  approaches  to  synthetic  vision  simply  use  color  to  fill  code 
areas,  a  common  approach  being  to  use  similar  colors  to  those  found  on  traditional  aeronautical 
charts  (e.g.,  blue  for  Class  B  controlled  airspace,  purple  for  Class  C  controlled  airspace). 
Detailed  images  equivalent  to  current  print  maps  and  charts  in  all  detail,  however,  require  color 
pixel  densities  of  over  200  per  inch,  or  twice  that  in  fielded  aircraft.  Another  synthetic  vision 
approach  uses  photorealistic  texture  maps  based  on  satellite  imagery  overlaid  on  underlying 
digital  terrain  elevation  data  to  provide  the  pilot  a  display  that  is  as  close  as  possible  to  what 
he/she  would  see  out  the  window  in  conditions  with  ceiling  and  visibility  unlimited.  This 
approach  is  somewhat  more  demanding  of  the  color  rendering  capability  of  the  display  used.  The 
grayscale  of  cockpit  displays  is  typically  16  to  256  levels  (4  to  8  bits  per  monochrome  pixel); 
however,  human  visual  capability,  real  world  scenes,  and  advanced  sensors  are  several  orders  of 
magnitude  more  detailed  (14-20  bits  per  monochrome  pixel).  The  sensors  research  community 
has  informed  the  display  research  community  that  higher  grayscale  performance  in  the  cockpit  is 
needed  for  future  systems.  Because  most  operations  are  purposefully  carried  out  in  relatively 
calm  air  and  most  flight  time  is  in  level  flight,  higher  resolution  displays  could  improve 
productivity  of  aircrews  by  enabling  advanced  synthetic  vision  formats. 

The  primary  challenge  in  selection  of  an  HDD  for  synthetic  vision  is  size:  the  bigger  the  better. 
In  contrast  to  a  traditional  attitude  indicator  display,  the  size  of  a  head-down  synthetic  vision 
display  is  directly  related  to  how  much  of  the  world  is  viewable  by  the  pilot.  What  avionics 
engineers  are  often  tempted  to  do,  usually  in  response  to  complaints  of  pilots  who  feel  they  are 
“looking  at  the  world  through  a  soda  straw”,  is  to  independently  manipulate  the  geometric  field 
of  view  (GFOV)  of  the  display  in  an  attempt  to  compensate  for  small  display  size.  With  respect 
to  GFOV,  synthetic  vision  systems  displaying  a  database-based  image  are  similar  to  systems 
displaying  a  sensor-based  image  (whether  a  single  sensor  or  a  fused  image  from  multiple 
sensors).  The  field  of  view  (FOV)  of  a  sensor  is  often  greater  than  the  FOV  provided  by  an  HDD 
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when  viewed  from  the  cockpit  design  eyepoint.  As  an  example,  suppose  the  maximum  FOV  of  a 
Forward-Looking  Infrared  (FLIR)  sensor  aboard  an  aircraft  is  25°.  The  most  desirable  approach 
is  to  include  a  display  in  the  cockpit  that  is  capable  of  displaying  the  entire  image  on  a  display 
that  also  has  a  FOV  of  25°  when  viewed  from  the  cockpit  design  eyepoint.  This  results  in  a 
cockpit  display  in  which  the  GFOV  of  the  image  displayed  matches  the  FOV  of  the  display 
hardware  and  the  overall  image  is  accurate:  neither  magnified  nor  minified.  This  approach  is 
shown  in  Figure  6. 


Figure  6.  Representation  of  a  forward-looking  infrared  sensor  image  from  a  25°  FOV  sensor  shown 

in  a  25°  FOV  display. 

Unfortunately,  the  cockpit  display  in  the  above  example  is  not  one  currently  found  in  any 
military  cockpit.  It  would  have  to  be  roughly  15-in.  wide  to  result  in  a  FOV  of  25°  at  a  viewing 
distance  of  30”. 

(FOV=  2(arctan(0.5(display  size)/viewing  distance))). 

Now  suppose  a  more  likely  scenario  in  which,  because  of  cost,  weight,  or  space  limitations,  the 
largest  HDD  available  to  the  cockpit  designer  is  a  6”  wide  multifunction  display  (MFD).  At  a 
design  eyepoint  of  30”,  the  FOV  of  the  MFD  -  the  angle  subtended  on  the  pilot’s  retina  -  would 
be  roughly  10°  (2(arctan(0.5(6”)/30”))  =  11.4°).  The  cockpit  designer  is  now  faced  with  a  choice: 
either  the  MFD  can  display  less  than  half  the  image  the  sensor  is  capable  of  showing,  or  the 
MFD  can  show  the  entire  sensor  image.  If  the  designer  chooses  the  former,  sensor  capability  is 
wasted  and  pilots  will  voice  the  familiar  “soda  straw”  complaint.  If  the  designer  chooses  the 
latter,  pilots  can  now  see  the  entire  sensor  image,  but  it  is  minified  by  a  factor  of  roughly  2.5  to 
1 .  These  two  approaches  are  illustrated  in  Figure  7. 
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FDV  ^  ^(y^  GFDV  ^  25^ 


FDv  =  ^(y^  QPOV  =  1(r 


Figure  7.  Illustration  of  two  design  choices  when  faced  with  a  mismatch  between  FOV  of  sensor  or 
synthetic  image  and  FOV  of  cockpit  display:  minify  the  image  and  allow  the  pilot  to  see  all  of  it 
(left),  or  discard  part  of  the  image  while  maintaining  accurate  scene  geometry  (right). 

Pilots  will  almost  always  choose  the  option  associated  with  a  higher  GFOV  because  it  allows  for 
greater  situation  awareness.  However,  the  ramifications  of  this  choice  go  well  beyond  situation 
awareness  and  consequences  can  be  severe.  Choosing  a  higher  GFOV  in  this  case  results  in 
minification  of  the  image  and  perceptual  distortions.  These  distortions,  of  depth  perception  and 
angular  perception  (especially  depression  and  glide-path  angles),  can  be  subtle  and  not  obvious 
unless  the  resultant  display  is  compared  side  by  side  with  a  non-minified  image.  The  distortion  of 
perceived  size  (minification)  is  associated  with  distortions  in  perceived  depth  (e.g.,  Bigham, 
2000)  and  has  been  shown  to  impact  performance  and  situation  awareness.  Research  has  shown 
that  people  adapt  to  these  types  of  images,  but  adaptation  takes  time  (several  minutes)  and  - 
most  importantly  -  it  is  only  partial:  some  distortion  of  perceived  distance  and  size  remains.  One 
can  imagine  the  impact  in  a  high-speed  low-level  flight  environment:  simply  look  at  the  images 
presented  above  and  estimate  for  yourself  the  size  and  distance  of  the  lead  vehicle  in  each  image 
and  the  glide-path  angle  to  the  road.  It  is  not  the  same  in  both  images. 

Until  and  unless  truly  panoramic  HDDs  are  included  in  the  cockpit,  pilots  will  be  forced  to  adjust 
to  any  mismatches  between  an  HDD’s  physical  FOV  from  the  design  eyepoint  and  the  GFOV  of 
the  image  contained  in  the  display.  However,  the  larger  the  HDD  that  a  cockpit  designer  has 
available,  the  smaller  will  be  any  mismatch  between  the  physical  FOV  of  the  display  and  the 
GFOV  of  the  synthetic  image  portrayed  —  and  the  less  adjustment  will  be  required  of  the  pilot. 
The  primary  consideration  in  selecting  an  HDD  for  synthetic  vision  is  maximizing  the  viewable 
area  of  the  display  (solid  angle  of  image  at  design  eye  point). 

HEAD-UP  DISPLAY  REQUIREMENTS 

The  primary  driver  for  including  a  HUD  or  HMD  in  the  cockpit  is  typically  the  desire  for  a 
pilot’s  eyes  and  attention  to  remain  focused  outside  the  cockpit  while  needed  information  is 
shown  on  the  HUD  or  HMD.  From  a  human  factors  perspective,  two  critical  factors  driving 
symbology  design  decisions  (including  any  HUD  or  HMD  synthetic  vision  symbology)  are  a 
pilot’s  need  to  see  through  the  display,  and  the  criticality  of  the  information  displayed.  As  a 
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general  rule,  the  number  of  pixels  in  the  HUD  or  HMD  should  be  the  absolute  minimum  needed 
for  the  pilot  to  safely  accomplish  the  mission.  Anything  more  than  this  minimum  is  usually 
perceived  as  clutter.  Further,  the  information  that  is  displayed  should  not  draw  pilots’  attention 
away  from  the  external  visual  scene.  Research  has  shown  that  the  more  conformal  a  HUD  or 
HMD  symbology  set  is  (i.e.,  the  more  the  symbols  in  the  HUD  or  HMD  conform  to  objects  in 
the  external  visual  scene  or  far  domain),  the  better  pilots  will  be  able  to  attend  simultaneously  to 
the  information  contained  in  the  symbology  and  events  in  the  external  visual  scene. 

In  the  case  of  a  HUD  or  HMD  synthetic  vision  display,  the  GFOV  of  the  synthetic  image  will  be 
identical  to  the  physical  FOV  of  the  display  from  the  design  eyepoint.  Any  mismatch  between 
these  two  fields  of  view  will  produce  misregistration  between  objects  in  the  synthetic  image  and 
objects  in  the  external  visual  scene  on  which  the  synthetic  image  is  overlaid.  While  this  may  be 
acceptable  in  complete  instrument  meteorological  conditions  (i.e.,  when  there  is  absolutely  no 
external  visual  scene  underlying  the  synthetic  imagery),  research  has  shown  that  misregistration 
between  synthetic  and  real  imagery  in  conditions  allowing  a  simultaneous  view  of  both  has  a 
significant  adverse  impact  on  subjective  ratings  of  the  usability  of  the  synthetic  imagery  and  pilot 
acceptance  of  the  resulting  display.  In  the  case  of  a  HUD  or  HMD,  the  reason  to  select  a  display 
with  the  largest  FOV  possible  is  not  to  minimize  perceptual  distortions  caused  by  a  GFOV/FOV 
mismatch,  but  because  an  increase  in  FOV  results  in  a  1-to-l  increase  in  GFOV  -  and  a  1-to-l 
increase  in  the  extent  of  the  synthetic  image  viewable  by  the  pilot. 

Perhaps  the  primary  requirements  of  a  HUD  or  HMD  used  as  a  synthetic  vision  display  are 
accuracy  (temporal  and  spatial  across  the  entire  system),  brightness,  and  the  capability  to 
simultaneously  display  multiple  image  sources.  The  display  designer  may  or  may  not  have  much 
direct  control  over  temporal  and  spatial  accuracy.  Processing  delays  in  avionics,  optical 
distortion  associated  with  the  windscreen  or  canopy,  and  exact  placement  of  the  pilot’s  eyes  will 
all  lead  to  spatial  and  temporal  inaccuracies  that  may  produce  misregistration  between  synthetic 
and  real  images.  However,  the  display  designer  should  take  whatever  steps  he/she  can  to  ensure 
that  the  display  optics  do  not  contribute  to  misregistration  and  that  the  pilot  (or  maintainer)  has 
the  capability  to  “boresight”  the  system  to  compensate  for  static  and  dynamic  sources  of  error. 

Brightness  is  always  important  in  HUDs  and  HMDs  because  they  must  be  readable  against 
extremely  bright  backgrounds:  primary  symbology  must  be  legible  against  a  glare  source  on  the 
order  of  10,000  fL.  Brightness  is  also  important  for  any  synthetic  image  underlying  this  primary 
symbology.  If  the  maximum  brightness  achievable  by  a  HUD  raster  is  only  50  fL  or  so,  then  the 
display  may  be  suitable  for  use  only  at  night.  While  synthetic  vision  has  been  proposed  as  a 
technology  that  is  most  useful  in  reduced-visibility  conditions,  reduced  visibility  is  not  always 
associated  with  reduced  illuminance.  One  example  of  this  is  flight  in  clouds  near  the  top  of  a 
cloud  layer.  Such  an  environment  could  easily  wash  out  a  synthetic  image  if  it  is  not  displayed 
with  sufficient  luminance.  Further,  there  are  some  occasions  when  pilots  might  find  synthetic 
vision  displays  useful  even  in  visual  meteorological  conditions  (VMC).  For  example,  mishaps 
have  occurred  in  VMC  on  approach  to  unfamiliar  snow-covered  airfields,  over  open  water,  and 
over  open  desert.  In  such  cases,  normal  depth  and  self-motion  cues  are  degraded  and  synthetic 
vision  could  be  a  useful  means  of  regaining  situation  awareness  lost  due  to  lack  of  these  cues  - 
but  only  if  the  synthetic  image  is  bright  enough  to  be  seen  in  these  conditions.  Finally,  synthetic 
vision  HUDs  and  HMDs  should  be  capable  of  simultaneously  displaying  two  image  sources,  as 
in  a  stroke-on-raster  HUD.  Most  current  synthetic  vision  concepts  call  for  primary  flight 
symbology  (i.e.,  critical  gauges  and  alphanumerics)  to  be  displayed  as  a  separate  entity  from  the 
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underlying  synthetic  image.  This  allows  declutter  and  brightness  controls  to  function 
independently  and  also  allows  the  symbology  designer  to  use  brightness  coding  to  perceptually 
separate  the  content  of  the  two  images. 
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III.  SYNTHETIC  VISION  FORAAAT  DEVELOPMENT 


The  National  Aeronautics  and  Space  Administration  (NASA)  Aviation  Safety  Program  (ASP) 
included  the  Synthetic  Vision  Systems  (SVS)  program,  the  largest  active  synthetic  vision 
research  and  development  effort  for  the  aviation  environment.  Comprised  of  NASA  personnel 
and  industry  partners,  the  SVS  team  conducted  tests  aboard  the  Airborne  Research  Integrated 
Experiments  System  (ARIES)  Boeing  757-200  aircraft  to  study  the  utility  of  a  system  for 
approach,  landing,  and  ground  operations.  During  trials  at  Dallas-Ft.  Worth  (DEW)  in  October  of 
2000,  NASA  and  Rockwell  Collins  tested  a  generic,  opaque  texture  map  format  and  a 
photorealistic,  opaque  texture  map  format  in  a  Flight  Dynamics  HUD.  All  testing  occurred  at 
night.  In  both  conditions,  pilots  had  the  option  to  declutter  the  HUD.  At  the  same  time,  these 
formats  were  also  tested  on  the  head-down  display  (HDD).  Additional  features  examined  on  the 
HDD  were  display  size  and  pilot  selectable  field-of-view.  Results  of  the  DFW  trials  based  on 
pilot  comments  showed  a  pilot  preference  for  photorealistic  terrain  over  the  generic  format  for 
judging  distances  and  closure  rates. 

Rockwell  Collins,  Flight  Dynamics,  and  Delft  University  are  also  working  together  to  produce 
synthetic  vision  systems  for  HUDs  and  HDDs,  see  figures  8  and  9. 


Figure  8.  Image  of  the  Head-Down  Display  Figure  9.  Image  of  the  Head-Up  Display 

symbology  and  the  texture  map  synthetic  symbology  developed  by  Rockwell  Collins, 

terrain  being  developed  by  Rockwell  Collins. 

The  goal  is  to  improve  pilot  awareness  of  both  navigational  and  terrain  features  by  using  a 
combination  of  Tunnel-in-the-Sky  and  database-derived  synthetic  terrain.  Head-down,  the 
synthetic  terrain  may  be  dynamically  color-coded  for  the  level  of  threat  it  represents  or  displayed 
as  a  texture  map.  Head-up,  synthetic  terrain  may  be  wire-frame,  polygons,  or  monochrome 
texture  maps  and  can  underlay  the  basic  HUD  symbology. 

Similar  efforts  by  TU-Darmstadt  and  Stanford  University  to  display  synthetic  imagery  on  head- 
down  and  head-mounted  displays  have  yielded  the  display  formats  illustrated  in  Figures  10  and 
11. 
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Figure  10.  Image  of  the  Head-Down  Display 
symbology  and  the  synthetic  vision  system 
being  developed  by  TU-Darmstadt. 


Figure  1 1 .  Image  of  the  Head-Down  Display 
symbology  and  the  synthetic  vision  system 
being  developed  by  Stanford  University. 


While  civilian  efforts  have  focused  on  terminal  operations,  military  organizations  are  also 
looking  at  other  tasks  that  require  aircraft  to  be  near  the  ground,  such  as  fixed  wing  low-level 
ingress  and  rotorcraft  operations.  At  AFRL’s  System  Control  Interfaces  Branch,  the  research 
focus  has  been  primarily  on  HUD  pathway  symbology  for  flight  guidance,  and  synthetic  terrain 
imagery  for  terrain  awareness.  HUDs  are  now  the  primary  flight  references  in  almost  all 
fighter/attack  aircraft,  the  Air  Force's  newer  transports,  and  are  being  installed  as  upgrades  to 
some  older  platforms.  Figures  12  and  13  provide  examples  of  the  symbology  tested  under 
simulated  Instrument  Meteorological  Conditions  (IMC)  using  a  monochrome  texture  map  and 
100-meter  grid,  respectively. 


Figure  1 2.  Pathway-in-the-sky  superimposed 
on  texture  format  synthetic  terrain. 


Figure  13.  Pathway-in-the-sky  superimposed 
on  grid  format  synthetic  terrain. 


In  the  AFRL  study,  pilots  preferred  the  grid  format  and  reported  better  situation  awareness  using 
this  format  even  though  the  texture  map  format  is  arguably  more  realistic  (Snow  &  Reising, 
1999).  Both  displays  are  conformal  and  the  only  difference  between  the  two  is  the  format  of  the 
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synthetic  terrain  imagery.  The  formats  above  were  tested  during  a  low-level  ingress  task.  The 
grid  format  will  also  be  tested  during  approach  and  landing  tasks  in  an  upcoming  study. 

In  related  research  and  under  the  auspices  of  the  Air  Force’s  Dual  Use  Science  &  Technology 
(DUS&T)  program,  AFRL  and  Rockwell  Collins  conducted  simulations  and  a  flight  test  of  head- 
up  symbology  for  synthetic  terrain  and  pathway  guidance  in  the  context  of  a  military  transport 
mission  in  2002. 
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IV.  SITUATION  AWARENESS 


As  discussed  previously,  situation  awareness  (SA)  is  a  critical  factor  in  the  safety  and 
survivability  of  USAF  aircrew.  Deficits  in  SA  can  have  profound  and  tragic  consequences,  an 
example  of  which  is  the  type  of  accident  known  as  Controlled  Flight  Into  Terrain  (CFIT).  From 
1987  through  1997,  the  USAF  suffered  190  fatalities  and  98  destroyed  aircraft  due  to  CFIT 
mishaps  (Moroze  &  Snow,  1999).  A  CFIT  accident  is  commonly  defined  as  one  in  which  an 
otherwise  serviceable  aircraft,  under  control  of  the  crew,  is  flown  (unintentionally)  into  terrain, 
obstacles  or  water,  with  no  prior  awareness  on  the  part  of  the  crew  of  the  impending  collision 
(Wiener,  1977).  While  warning  systems  have  reduced  the  rate  of  CFIT  accidents,  this  type  of 
solution  focuses  on  a  symptom:  that  collision  is  impending.  To  eliminate  CFIT,  we  must  also 
focus  on  the  underlying  cause:  pilots  lose  situation  awareness  concerning  the  relationship 
between  their  flight  paths  and  the  surrounding  terrain.  One  method  of  doing  this  is  to  include  a 
display  of  flight  path  and  surrounding  terrain  in  the  pilot’s  primary  flight  reference,  several 
examples  of  which  were  shown  in  previous  sections.  The  purpose  of  this  section  is  to  compare 
two  situation  awareness  metrics  used  in  the  present  research,  provide  lessons  learned  in  their 
implementation,  and  examine  the  relationship  between  SA  and  performance. 

A  number  of  definitions  and  metrics  have  been  proposed  for  SA.  Among  the  subjective  SA 
metrics  proposed  is  the  Situation  Awareness  adaptation  of  the  Subjective  Workload  Dominance 
technique  (SA-SWORD)(Vidulich  &  Hughes,  1991).  This  technique  uses  a  paired-comparison 
form  presented  shortly  after  testing  to  elicit  relative  rankings  of  all  conditions  on  SA.  Vidulich 
and  Hughes  (1991)  highlight  the  importance  of  defining  SA  for  observers  performing  the  paired 
comparisons  to  ensure  that  comparisons  are  made  on  roughly  the  same  underlying  basis  and 
results  then  generalize  across  observers.  The  definition  given  subjects  in  the  present  research  was 
based  on  that  of  Endsley  (1995b)  and  designed  to  encompass  the  three  levels  of  SA  that  she 
proposes:  “Perception  of  elements  in  the  environment,  comprehension  of  their  meaning,  and 
projection  of  future  status.’’ 

The  only  objective  SA  measure  in  widespread  use  today  is  the  Situation  Awareness  Global 
Assessment  Technique  (SAGAT)  proposed  and  tested  by  Endsley  (1995a).  SAGAT  is  a  method 
in  which  the  simulation  or  test  is  briefly  halted  to  ask  pilots  SA-relevant  questions.  The  accuracy 
with  which  pilots  answer  these  questions  is  then  the  measure  of  their  SA.  Because  this  technique 
involves  interruption  of  the  task  in  which  SA  is  measured,  there  is  cause  for  concern  about  the 
possible  intrusiveness  of  the  technique.  Interrupting  an  operator’s  task  has  the  potential  to 
interrupt  the  operator’s  working  memory,  attention  state,  and  mental  model  of  the  task.  Endsley 
(1995a),  using  a  relatively  global  performance  measure,  reports  finding  no  effect  of  SAGAT 
interruptions.  The  design  of  the  current  experiment  was  set  up,  in  part,  to  directly  gauge  the 
intrusiveness  of  the  interruptions  associated  with  SAGAT  use. 

With  regard  to  correlation  between  subjective  and  objective  SA  measures,  the  evidence  so  far 
leads  to  the  conclusion  that  there  isn’t  any  (Boag,  Neale,  &  Neal,  1999;  Endsley,  Selcon, 
Hardiman,  &  Croft,  1998).  One  explanation  for  this  may  be  that  operators  essentially  “don’t 
know  what  they  don’t  know.’’  Endsley  et  al.  (1998)  have  proposed  that  the  dissociation  is  due  to 
subjective  SA  measurement  being  based  on  confidence  in  perceived  SA  and  conclude  that  it 
therefore,  “casts  some  doubt  on  the  validity  of  subjective  SA  measurements  as  an  indication  of  a 
person’s  actual  SA,”  (Endsley  et  ah,  1998  p.  86).  However,  in  the  studies  conducted  by  Endsley 
et  al.  and  Boag  et  ah,  the  measures  compared  were  the  Situational  Awareness  Rating  Technique 
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(SART)(Taylor,  1990)  and  SAGAT.  SART  is  a  subjective  technique  in  which  a  score  is  obtained 
from  a  combination  of  operator  ratings  of  their  own  understanding  of  the  situation,  the  demand 
imposed  on  them,  and  the  supply  available  to  meet  the  demand.  The  underlying  construct  or 
definition  of  SA  upon  which  this  technique  is  based  (perceived  understanding  minus  the  degree 
to  which  task  demand  exceeds  operator  capability)  seems  fundamentally  different  from  the  three 
levels  of  SA  proposed  by  Endsley.  Based  on  the  underlying  constructs  measured  by  the  two 
techniques,  it  does  not  seem  surprising  that  a  correlation  between  SAGAT  and  SART  has  not 
been  found.  Should  this  be  interpreted  as  a  difference  between  objective  and  subjective  SA 
measures  or  as  a  difference  between  SA  measurement  techniques  that  measure  fundamentally 
different  aspects  of  SA?  If  one  were  able  to  compare  SAGAT  results  with  a  subjective  SA 
measure  that  is  based  upon  the  same  theoretical  construct  of  SA  used  to  generate  SAGAT 
questions,  would  one  still  find  no  correlation  between  objective  and  subjective  SA  measures?  It 
is  our  contention  that  by  having  subjects  rate  S A  using  SA-SWORD  -  and  basing  their  ratings  on 
the  same  definition  of  SA  used  to  generate  SAGAT  questions  -  the  current  study  is  perhaps  the 
first  to  achieve  a  true  “apples-to-apples”  comparison  of  objective  and  subjective  SA  measures. 
This  is  crucial  to  assessing  the  validity  of  subjective  SA  measurement  and  to  discovering  whether 
the  basis  for  dissociations  between  SA  measurement  techniques  found  so  far  has  been  operators’ 
inaccurate  perception  of  their  SA,  or  if  it  has  been,  instead,  differences  in  the  underlying 
phenomenon  being  measured. 

METHOD 

Participants 

Twelve  pilots  participated  in  this  study.  Subject  pilots  were  required  to  have  experience  in  a 
military,  HUD-equipped,  fighter/attack  aircraft.  All  were  male  with  military  flying  experience 
ranging  from  1,300  to  3,775  hours,  with  an  average  of  2,634  hours. 

Experimental  Design 

The  study  used  a  2  x  4  x  2  mixed-factors  design  with  two  within- subjects  factors  and  one 
between-subjects  factor.  The  first  within- subjects  factor  was  visibility  (Day  IMG,  or  Instrument 
Meteorological  Conditions,  and  Night  IMC).  The  second  within-subjects  factor  was  synthetic 
terrain  type  (none,  grid,  partial  grid,  and  texture-map).  The  between-subjects  factor  was  type  of 
situation  awareness  measurement  (SA-SWORD  only  or  SA-SWORD  and  SAGAT). 

Procedure 

Pilots  received  training  in  both  the  classroom  and  in  the  cockpit.  Classroom  training  included 
briefing  the  subjects  on  the  purpose  of  the  study,  describing  the  different  experimental 
conditions,  and  explaining  the  desired  response  to  TEWS  alerts:  dodging  left  or  right  off  the  path 
and  “into  the  weeds”  (i.e.,  going  as  low  as  they  could  to  get  under  the  radar)  as  soon  as  possible. 
Cockpit  training  consisted  of  cockpit  familiarization  and  flying  a  practice  profile  to  demonstrate 
the  pathway  symbology  and  allow  the  pilot  to  become  familiar  with  the  aeromodel. 

Subjects  flew  eight  different  ingress  scenarios  (all  MK-84  deliveries).  Each  ingress  scenario 
contained  two  pop-up  SAM  threats,  one  located  near  the  beginning  of  the  scenario  and  one 
located  near  the  end  of  the  scenario.  These  pop-up  SAM  threats  were  designed  to  force  pilots  off 
the  path  so  that  data  could  be  collected  in  the  absence  of  the  pathway.  Pilots  continued  to  fly  for 
fifteen  seconds  after  a  TEWS  alert  sounded,  after  which  the  simulation  was  briefly  interrupted 
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and  then  restarted  with  the  pilot  back  on  the  ingress  path.  Subjects  flew  a  short  practice  trial  in 
each  experimental  condition  immediately  prior  to  flying  the  ingress  scenario  in  that  condition. 

Pilots  in  the  SAGAT  plus  SA-SWORD  condition  answered  a  total  of  eighteen  questions  during 
each  ingress  scenario,  half  (selected  at  random)  during  an  on-path  simulation  interruption  and 
half  after  one  of  the  off-path  simulation  interruptions  that  followed  each  SAM  alert.  Subject  to 
these  restrictions,  interruptions  were  placed  at  random  within  the  ingress  scenario.  A  sample  of 
the  eighteen  questions  asked  of  each  pilot  in  the  SAGAT  plus  SA-SWORD  condition  and  their 
associated  dependent  measures  are  shown  in  Table  1 . 

Presentation  of  the  ingress  scenarios  and  the  eight  combinations  of  weather  and  synthetic  terrain 
were  counterbalanced  using  a  standard  Latin  square.  Odd-numbered  subjects  were  assigned  to 
the  SA-SWORD  only  condition,  while  even-numbered  subjects  were  assigned  to  the  SAGAT  and 
SA-SWORD  condition. 

Apparatus 

This  study  was  conducted  in  a  single-seat,  fixed-base  fighter  cockpit  simulator.  Head-down 
displays  included  a  moving  map  showing  a  top-down  view  of  the  terrain  database  and  a  Threat 
Electronic  Warfare  System  (TEWS)  display  to  alert  pilots  of  lock-on  by  surface-to-air  missiles 
(SAMs).  A  50°  horizontal  x  40°  vertical  field  of  view  (FOV)  out-the-window  scene  was 
projected  on  a  screen  in  front  of  the  simulator.  A  30°  horizontal  x  20°  vertical  FOV  HUD  was 
simultaneously  projected  on  the  out-the-window  screen.  The  cockpit  also  included  a  sound 
system  to  generate  engine  noise  and  auditory  events. 

RESULTS 

Performance 

A  MANOVA  conducted  on  the  three  primary  flight  performance  measures  (RMS  error  in 
airspeed,  lateral ,  and  vertical  deviation)  revealed  no  statistically  significant  effect  of  the  type  of 
SA  measurement  used  (a  <  0.2).  The  observed  power  (1  -  P)  of  this  test  was  0.27.  Similarly,  no 
statistically  significant  effects  were  found  in  univariate  tests  (ANOVAs)  for  this  variable  on  Gs 
pulled  while  off-path  (a  <  0.2,  1  -  p  =  0.29),  MK-84  delivery  error  (a  <  0.2,  1  -  p  =  0.26),  or 
reaction  time  to  SAM  alerts  (a  <  0.2,  1  -  p  =  0.26). 

Situation  Awareness 

In  contrast  to  performance  measures,  analysis  of  SA-SWORD  data  revealed  significant  main 
effects  for  both  synthetic  terrain  format  (F(3,  30)  =  26.43,  p  <  0.001)  and  visibility  (F(l,  10)  = 
5.61,  p  <  0.05).  These  are  illustrated  in  Figure  14.  The  only  SAGAT  question  for  which  a 
statistically  significant  effect  was  found  was  Question  7  (see  Table  1).  The  effect  of  synthetic 
terrain  on  error  in  answering  this  question  was  significant  (F(3,  15)  =  4.13,  p  <  0.05)  as  shown  in 
Figure  15. 
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Table  1.  Sample  SAGAT  questions  used  in  the  study. 


SAGAT  Question 

Dependent 

Measure 

1 .  Estimate  your  pitch. 

Error  in 
degrees 

7.  What  direction,  using  the  hours 
of  a  clock,  is  the  nearest  terrain  in 
front  of  you  that  is  above  your 
current  altitude? 

Error  in 
increments 
off  from 
correct 

10.  Estimate  how  much  you  are 
currently  above  or  below  your 
commanded  airspeed. 

Error  in 
knots 

12.  From  your  current  position, 
what  is  the  best  escape  route  to 
avoid  terrain? 

Correct  / 
Incorrect 

13.  Give  a  detailed  description  of 
the  terrain  at  10  o’clock,  halfway 
to  the  horizon. 

Correct  / 
Incorrect 

SAGAT  Questions  13-18  essentially  asked  the  same  question  (describe  terrain),  but  regarding 
various  distances  and  directions.  The  number  of  correct  answers  to  these  questions  was  therefore 
also  analyzed  as  a  separate  dependent  variable,  but  no  statistically  significant  (a  <  0.05)  effects 
were  found. 
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Figure  14.  Effect  of  synthetic  terrain  format 
and  visibility  on  SA-SWORD  ratings  (error  bars 
are  90%  confidence  intervals). 


Figure  15.  Effect  of  synthetic  terrain  format 
on  error  in  answering  SAGAT  Question  7 
(error  bars  are  90%  confidence  intervals). 
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Concerning  the  relationship  between  SA-SWORD  ratings  and  performance  in  answering  SAGAT 
questions,  correlations  between  SA-SWORD  ratings  and  magnitude  of  error  for  SAGAT 
Questions  1-3,  5,  and  7-10  (i.e.,  those  for  which  numeric  error  scores  were  obtained  rather  than 
scores  of  correct/incorrect)  were  low  (absolute  values  ranging  from  0.04  to  0.15)  and  none  were 
statistically  significant  (a  <  0.05).  Similarly,  no  significant  correlation  was  found  between  SA- 
SWORD  rating  and  number  of  errors  on  SAGAT  terrain  description  questions  (i.e..  Questions 
13-18). 

With  regard  to  intrusiveness  of  SAGAT  interruptions,  the  post-session  questionnaire  asked 
subjects,  “How  did  you  feel  about  the  intrusiveness  of  stopping  the  study  to  ask  situation 
awareness  questions?”  They  were  asked  to  rate  intrusiveness  on  a  1  to  7  scale,  with  7  being 
greatest  intrusiveness.  Five  out  of  six  subjects  responded  with  a  rating  of  4,  or  neutral;  the  fifth 
responded  with  a  6. 

DISCUSSION 

While  the  results  should  be  interpreted  cautiously  because  of  the  small  sample  size  (n  of  6  in  the 
relevant  cells  for  the  between-subjects  comparison),  they  indicate  no  strong  relationship  between 
SA  and  any  of  the  performance  measures  used.  This  was  true  not  just  for  flight  performance 
measures,  but  for  measures  (e.g.,  reaction  time  to  a  SAM  alert)  that  might  be  expected  to  be 
more  dependent  upon  SA.  The  finding  of  large  differences  in  SA-SWORD  ratings  between 
experimental  conditions,  without  associated  performance  differences  suggests  that  the 
relationship  between  performance  and  subjectively  perceived  SA  may  be  especially  tenuous. 
Like  those  of  previous  studies  (Boag  et  ah,  1999;  Endsley  et  ah,  1998),  the  results  of  this  study 
lead  to  the  conclusion  that  there  is  not  a  strong  relationship  between  subjective  (i.e.,  SA- 
SWORD)  and  objective  (i.e.,  SAGAT)  measures  of  situation  awareness  -  even  when  the 
definition  and  dimensions  of  SA  used  as  the  basis  for  each  technique  are  identical. 

Finally,  these  results  also  support  the  hypothesis  that  task  interruptions  associated  with  SAGAT 
do  not  affect  performance.  In  written  comments,  two  pilots  indicated  specifically  that  they  felt 
the  interruptions  did  not  affect  their  flying  performance.  However,  pilot  comments  also  indicated 
that  their  attentional  or  cognitive  behavior  may  have  been  altered  by  the  fact  that  questions  were 
asked.  In  the  words  of  one,  “As  the  study  went  along  it  became  a  game  to  see  how  many  details 
of  the  maneuver  you  could  remember.”  In  the  words  of  another,  the  questions,  “helped  force  an 
increase  in  maintaining  a  good  scan  of  all  the  data,  knowing  I  might  have  to  recite  specific 
parameters.” 

Lessons  Learned 

Both  SA  measurement  techniques  have  advantages  and  disadvantages.  SA-SWORD  has  global 
sensitivity,  but  not  the  diagnosticity  of  SAGAT.  Also,  as  has  been  pointed  out  previously, 
operators  don’t  know  what  they  don’t  know  and,  like  most  subjective  techniques,  there  is  the 
possibility  of  halo  effects  from  performance  level  or  other  intervening  variables.  In  the  current 
study,  we  wished  to  measure  SA  both  on-path  and  off-path.  The  SA-SWORD  technique  is  not 
designed  to  do  this,  but  SAGAT  was  readily  adaptable  to  this  and  is,  in  general,  suitable  for 
measuring  SA  during  different  within-scenario  conditions,  although  this  can  make  random  timing 
of  SAGAT  interruptions  more  difficult. 
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While  it  has  good  diagnosticity,  SAGAT  results  in  multiple  variables  (the  error  on  each  question) 
rather  than  a  single  SA  score,  and  combination  of  these  variables  into  an  overall  measure  of  SA 
can  be  problematic.  Also,  instead  of  interval-scale  data,  SAGAT  questions  often  yield 
correct/incorrect  data,  lessening  the  power  of  statistical  analysis  techniques  that  may  be  applied. 
Finally,  SAGAT  is  only  as  good  as  the  questions  asked.  Good  questions  require  a  task  analysis, 
pilot  testing  to  ensure  that  questions  are  relevant  at  each  interruption  and  that  subjects  respond 
appropriately,  and  reliance  on  subject-matter  experts  for  questions  and  often  for  scoring.  In  our 
case,  correct  answers  to  several  of  our  most  interesting  SAGAT  questions  could  only  be  gleaned 
from  screen  captures  of  the  scenario  after  data  collection.  This  was  time-consuming  and 
sometimes  imprecise.  It  was  our  experience  that  the  SAGAT  technique  is  associated  with 
substantially  greater  overhead  than  SA-SWORD  in  terms  of  simulation  preparation,  data 
collection,  and  data  analysis.  However,  lack  of  intrusion  on  performance,  the  face  validity  of  the 
technique,  and  it’s  usefulness  as  a  diagnostic  tool  lead  us  to  conclude  that  it  is  a  valuable 
technique  and  one  that  we  will  use  in  the  future  -  in  conjunction  with  a  subjective  measure  of 
SA. 
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V.  EFFECTS  OF  PRIAAARY  FLIGHT  SYMBOLOGY  ON  WORKLOAD  AND  SITUATION 
AWARENESS  IN  A  HEAD-UP  SYNTHETIC  VISION  DISPLAY 

This  section  reports  the  results  of  a  study  in  which  HUD-experienced  pilots  flew  simulated 
complex  precision  approaches  to  landing  in  three  visibility  conditions,  with  and  without 
synthetic  terrain,  using  either  pathway-in-the-sky  symbology  or  more  traditional  military 
standard  HUD  symbology.  Workload  and  situation  awareness  measures  were  collected  to 
determine  the  relative  workload  associated  with  these  conditions  and  if,  as  has  been  proposed 
elsewhere,  flying  a  pathway-in-the-sky  display  is  associated  with  “cognitive  capture”,  or  a 
decrease  in  situation  awareness  concerning  things  other  than  the  pathway.  It  was  hypothesized 
that  including  pathway  and  synthetic  terrain  in  a  head-up  primary  flight  display  would  result  in  a 
conformal  symbology  set  that  naturally  draws  pilots’  attention  to  external  events.  It  was  also 
hypothesized  that  workload  could  be  reduced  by  allowing  pilots  to  maintain  spatial  orientation 
via  preattentive  processes  rather  than  relying  on  instruments  requiring  focal  vision  and  active 
interpretation. 

Controlled  Flight  Into  Terrain  (CFIT)  accidents  continue  to  be  a  major  source  of  fatalities  and 
airframe  losses  in  both  military  and  civil  aviation,  despite  on-board  warning  systems  (Moroze  & 
Snow,  1999;  Scott,  1996).  Examination  of  evidence  from  the  USAF  Safety  Center  reveals  that  a 
causal  factor  in  over  half  of  these  accidents  is  related  to  some  deficit  in  situation  awareness  (SA) 
(Moroze  &  Snow,  1999).  Current  on-board  warning  systems  (e.g.,  the  Ground  Proximity 
Warning  System)  have  reduced  CFIT  dramatically,  but  are  designed  to  prevent  disaster  once  it 
becomes  imminent  (i.e.,  once  the  system  detects  that  an  aircraft  flight  path  will  be  below  some 
minimum  safe  altitude).  What  are  needed  are  systems  to  improve  pilot  SA  to  the  point  where  the 
need  for  warnings  of  imminent  disaster  are  greatly  reduced  or  even  eliminated.  Both  enhanced 
(sensor-driven)  and  synthetic  (database-driven)  vision  systems  have  been  proposed  as  means  to 
enhance  pilot  SA  and  reduce  accidents  caused  by  SA  deficits.  Synthetic  vision  systems  have 
several  features  that  strictly  sensor-based  systems  do  not.  Among  these  features  are  infinite  field 
of  regard,  field  of  view  limited  only  by  display  hardware,  range  up  to  (and  even  beyond)  the 
visible  horizon,  and  ability  to  portray  surrounding  terrain  regardless  of  atmospheric  conditions. 
Synthetic  vision  systems  are  only  as  reliable  as  the  database,  navigation,  and  display  subsystems 
upon  which  they  are  built,  but  even  with  these  limitations  they  seem  a  useful  adjunct  to 
traditional  navigation  aids  and  sensor  systems.  Research  in  the  Air  Force  Research  Laboratory 
has  demonstrated  substantial  increases  in  pilot  SA  with  the  addition  of  synthetic  terrain  to  a 
simulated  HUD  (head-up  display)  (Snow  &  Reising,  1999).  Further,  this  increase  in  SA  was 
associated  with  a  reduction  in  ground  impacts  during  the  low-level  ingress  scenarios  used  in 
these  simulations.  An  example  of  the  synthetic  terrain  used  in  this  study  is  shown  in  Figure  16. 
The  grid  format  shown  in  this  figure  was  associated  with  the  largest  increase  in  SA  overall  (i.e., 
in  Instrument  Meteorological  Conditions  (IMC)  both  night  and  day)  and  is  the  format  that  was 
used  in  the  current  study.  This  format  is  also  easily  distinguishable  from  sensor  imagery  and 
therefore  suitable  for  use  in  displays  combining  sensor  and  database  imagery. 
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Figure  16.  Grid  format  synthetic  terrain. 


Previous  research  in  this  same  laboratory  (Reising,  et  al.,1995)  has  shown  that  a  pathway-in-the- 
sky  similar  to  the  Command  Flight  Path  Display  tested  in  the  1980s  (Hoover,  Cronauer,  & 
Shelley,  1985;  Dittenhauser,  Eulrich,  Reynolds,  1983)  allows  pilots  to  fly  complex  precision 
approaches  in  IMC  at  night  roughly  twice  as  well  compared  to  the  current  military  standard 
head-up  display  (MIL-STD  HUD)  found  in  MIL-STD  1787C  (DoD,  2000).  Pilots  using  the 
pathway  were  able  to  maintain  the  commanded  path  with  roughly  half  the  error  in  airspeed, 
lateral,  and  vertical  deviation.  Such  complex  approaches,  with  multiple  curves  and  descent  rates, 
may  become  common  in  the  next  decade  as  augmentation  systems  to  the  current  GPS  signal, 
both  military  and  civilian,  become  operational.  Later  research  showed  that  the  benefit  of  a 
pathway  HUD  in  landing  these  approaches  did  not  vary  with  visibility  in  three  daylight 
conditions,  but  did  not  directly  compare  the  pathway  to  the  MIL-STD  HUD  (Reising,  et  al., 
1998).  While  clear  differences  have  been  found  between  pathway  and  MIL-STD  HUD 
symbology  with  regard  to  performance  in  flying  and  landing  complex  approaches,  the 
differential  effects  of  these  two  symbology  sets  on  situation  awareness  and  workload  have  yet  to 
be  measured.  Measurement  of  SA  and  workload  is  especially  important  in  the  current  context  for 
two  reasons:  1)  control/response  ratio  considerations,  and,  2)  attention  management  concerns. 

Control/response  (C/R)  ratio  refers  to  the  ratio  of  control  movement  needed  to  achieve  a  given 
system  or  display  response  (Sanders  &  McCormick,  1987).  A  low  C/R  ratio  implies  a  great  deal 
of  display  movement  with  little  control  movement  (i.e.,  high  gain)  while  a  high  C/R  ratios 
implies  the  opposite  (i.e.,  low  gain).  The  C/R  ratio  for  a  pathway  display  is  essentially 
determined  by  the  width  of  the  path  displayed.  Current  GPS  approaches  call  for  tolerances  as 
tight  as  ±0.15  nautical  miles  or  roughly  1,800  feet.  However,  path  dimensions  in  the  literature  on 
this  topic  vary  widely.  Theunissen  (1997)  has  tested  tunnels  of  roughly  75,  150,  and  300  feet  in 
width.  Snow  and  Reising  (1999)  used  a  path  width  of  400  feet.  Williams  (2002)  used  a  path 
width  of  600  feet,  as  did  the  current  study,  roughly  simulating  an  approach  with  a  Required 
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Navigation  Performance  (RNP)  of  0.05  nautical  miles.  Several  studies  have  shown  that  path  and 
tunnel  displays  result  in  less  flight  technical  error  relative  to  traditional  flight  displays,  but  the 
question  becomes,  “At  what  cost?”  Is  this  increased  precision  associated  with  higher  workload? 
How  tight  a  tightrope  should  pilots  have  to  walk?  Putting  a  path  or  tunnel  display  in  the  cockpit 
is  likely  to  be  counterproductive  if  it  means  that  the  pilot  is  so  busy  maintaining  the  path  that  no 
time  or  attention  can  be  spared  for  anything  else.  It  is  critical  that  flight  technical  error,  situation 
awareness,  and  workload  be  measured  in  conjunction. 

With  regard  to  attention  management,  there  is  reason  to  be  concerned  about  cognitive  capture  or 
attentional  tunneling:  the  possibility  that  a  synthetic  vision  display,  especially  one  including  a 
pathway  or  tunnel,  will  be  so  compelling  and  contain  such  a  large  proportion  of  the  pilot’s 
information  requirements  that  awareness  of  other  displays  and  events  will  deteriorate  in 
comparison.  Some  authors  report  no  difference  in  situation  awareness  on-path  versus  off-path 
(Snow  &  Reising,  1999),  while  others  report  a  decrease  in  awareness  of  events  and  information 
not  contained  in  the  synthetic  vision  display  (Williams,  2002;  Olmos  &  Wickens,  1997).  To  date, 
most  of  these  studies  have  been  done  with  head-down  displays  and  the  differences  found  (or  not 
found)  seem  to  be  highly  task-dependent  (but  see  Fadden,  Ververs,  &  Wickens,  2001).  One 
purpose  of  the  current  study  was  to  see  whether  using  a  head-up  display  for  synthetic  vision,  and 
presumably  focusing  pilots’  attention  head-up  rather  than  head-down  would  alleviate  such 
concerns. 

METHOD 

Participants 

Thirteen  pilots  volunteered  to  participate  in  the  study.  All  were  Air  Force  pilots  with  HUD 
experience.  Pilot  experience  ranged  from  1700  hours  to  15000  hours  with  an  average  of  4819 
hours.  All  pilots  were  male.  Pilots  ranged  in  age  from  30  to  53  with  an  average  of  39. 

Experimental  Design 

The  study  used  a  3  x  2  x  2  within-subjects  design.  However,  there  were  only  ten  experimental 
conditions:  no  trials  were  run  in  the  VMC  Day  condition  with  synthetic  terrain. 

Independent  Variables.  The  independent  variables  manipulated  in  the  study  were,  1)  visibility 
condition  (Visual  Meteorological  Conditions  (VMC)  Day,  VMC  Night,  IMC  Day),  2)  primary 
flight  display  (MIL-STD  HUD  vs.  Pathway),  and,  3)  synthetic  terrain  (Grid  vs.  None).  The  IMC 
Day  condition  consisted  of  %-mile  visibility  with  a  100-foot  ceiling  (equivalent  to  ITS  CAT  II). 
Figures  17  through  19  show  the  MIL-STD  HUD  in  VMC  Day,  the  Pathway  in  IMC  Day,  and  the 
MIL-STD  HUD  in  VMC  Night  with  synthetic  terrain,  respectively.  These  figures  show  each 
symbology  set  from  the  same  vantage  point  (short  final)  and  aircraft  state. 
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Figure  17.  MIL-STD  HUD  in  VMC  Day. 


Figure  18.  Pathway  in  IMC  Day. 


Figure  19.  MIL-STD  HUD  in  VMC  Night  with  synthetic  terrain. 

Dependent  Variables.  The  dependent  variables  included  flight  technical  error  (lateral,  vertical, 
and  airspeed  deviation)  and  situation  awareness  and  workload  measures.  The  two  situation 
awareness  measures  used  were  the  Situation  Awareness  Global  Assessment  Technique  (SAGAT) 
and  the  Situation  Awareness  adaptation  of  the  Subjective  Workload  Dominance  technique  (SA- 
SWORD).  The  former  is  an  objective  measure  based  on  responses  to  task-relevant  questions 
(Endsley,  1995)  while  the  latter  is  a  subjective  paired-comparison  technique  (Vidulich,  Ward,  & 
Schueren,  1991).  The  SAGAT  questions  asked  in  the  study  are  listed  in  Table  2. 

The  two  workload  measures  used  were  NASA  TLX  (NASA  Task  Load  Index)  and  SWORD 
(Subjective  Workload  Dominance  technique).  The  former  is  a  rating  technique  with  six  subscales 
while  the  latter  is  a  paired-comparison  technique.  Responses  to  SAGAT  questions  and  NASA 
TLX  ratings  were  taken  during  each  approach  while  SWORD  and  S A- SWORD  measures  were 
taken  at  the  conclusion  of  the  experiment. 
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Table  2.  SAGAT  questions  asked. 


1 .  Estimate  your  pitch. 


2.  Estimate  your  indicated  airspeed. 


3.  Estimate  your  altitude  AGE. 


4.  Estimate  your  barometric  altitude. 


5.  Estimate  your  current  bank  angle. 


6.  Estimate  the  distance  to  your  destination. 


7.  Estimate  the  bearing  to  your  destination. 


8.  Estimate  the  number  of  your  next  waypoint. 


9.  Estimate  your  current  heading. 


10.  Estimate  your  current  descent  angle. 


1 1 .  Estimate  your  vertical  velocity. 


12.  The  terrain  at  12  o'clock,  midway  to  the 
horizon  is:  rising,  descending,  roughly  flat,  or 
water? 

13.  The  terrain  at  12  o'clock,  at  the  horizon  is: 
rising,  descending,  roughly  flat,  or  water? 


14.  Does  your  CDM  currently  intersect  sky, 
terrain,  water,  or  the  runway? 

15.  Estimate  your  drift  angle. 


16.  Are  you  currently  accelerating,  decelerating, 
or  neither? 

17.  Estimate  the  bearing  to  the  nearest  terrain 
that  is  above  your  current  altitude. 

18.  Estimate  how  much  you  are  currently  above 
or  below  your  commanded  altitude. 

19.  Estimate  how  far  left  or  right  you  are  from 
the  centerline  of  your  commanded  path. 

20.  What  will  your  commanded  altitude  be  ten 
seconds  from  now? 

2 1 .  Estimate  the  descent  angle  of  your 
commanded  flight  path  ten  seconds  from 
now. 

22.  Estimate  your  commanded  heading  ten 
seconds  from  now. 

23.  What  direction,  if  any,  will  your  commanded 
flight  path  turn  in  the  next  ten  seconds  (left, 
right,  or  none)? 

24.  Locate  all  traffic,  ground  and  air,  currently 
present  in  the  environment. 

25.  Estimate  distance  to  the  nearest  traffic. 


26.  Estimate  the  bearing  to  the  nearest  traffic. 


Procedure 

Participants  received  an  introductory  brief,  simulator  and  symbology  familiarization,  and  then 
flew  ten  practice  approaches,  once  in  each  experimental  condition.  Data  collection  then 
proceeded  with  participants  flying  ten  different  approaches  twice  each  (a  total  of  twenty 
approaches  during  the  data  collection  phase).  Each  experimental  condition  was  replicated  twice 
in  random  order.  The  simulation  was  briefly  interrupted  once  at  a  random  interval  during  each 
approach  to  ask  SAGAT  questions  and  take  NASA  TLX  ratings.  Two  to  four  F-16s  were  present 
in  the  airport  environment  during  each  approach  with  the  final  (twentieth)  approach  differing 
from  the  rest  in  that  one  of  the  F-16s  was  stationed  on  the  runway  at  the  touchdown  point.  This 
last  approach  was  always  conducted  in  one  of  the  two  VMC  conditions.  Upon  completion  of  all 
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approaches,  participants  rated  their  workload  and  situation  awareness  using  the  SWORD  and 
SA-SWORD  techniques,  and  filled  out  a  questionnaire  to  solicit  subjective  opinions  concerning 
the  conditions  and  symbology  sets  flown. 

Apparatus 

The  cockpit  used  was  a  fixed-based  simulation  of  a  generic  fighter/attack  aircraft  using  an  F-16 
aeromodel.  Primary  control  inputs  consisted  of  throttle,  force  stick,  touch  screen,  and  gear 
handle.  The  out-the-window  scene  was  viewed  on  three  projectors  surrounding  the  cockpit  with  a 
total  lateral  field  of  view  of  1 10°  and  a  vertical  field  of  view  of  30°.  HUD  symbology  was 
superimposed  on  the  center  projection  screen  with  a  field  of  view  of  30°  lateral  by  20°  vertical.  A 
picture  of  the  simulator  is  shown  in  Figure  20.  Head-down  instruments  included  a  moving  map 
and  a  traditional  primary  flight  display  (Attitude  Director  Indicator,  Horizontal  Situation 
Indicator,  Airspeed  and  Altitude  Indicators).  Participants  were  given  an  instrument  approach 
procedure  to  be  studied  prior  to  each  approach,  which  they  then  placed  on  a  kneeboard  for 
reference  throughout  the  approach.  A  sample  approach  procedure  is  shown  in  Figure  21. 


Figure  20.  Cockpit  simulator  used  In  the  study. 


RESULTS 

Flight  Technical  Error  (FTE) 

Flight  Technical  Error  data  collected  were  the  airspeed,  lateral,  and  vertical  deviations  from 
commanded  values.  These  data  were  then  broken  into  two  groups  based  on  occurrence  of  a 
secondary  task:  non-distracter  and  distracter.  Root  Mean  Square  Error  (RMSE)  was  calculated 
for  statistical  analysis.  Also  calculated  from  the  raw  FTE  data  were  the  percentages  of  time  spent 
outside  the  commanded  corridor  vertically  and  laterally. 

All  of  the  non-distracter  FTE  means  were  significantly  lower  (a  =  .05)  for  the  pathway  condition. 
Figure  22  shows  the  means  for  lateral  deviation  RMSE  and  is  representative  of  the  other  FTE 
measures  (error  bars  in  all  figures  represent  90%  confidence  intervals). 
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Figure  21.  Sample  instrument  approach  procedure  used  in  the  study. 


Percent  time  spent  off  path  was  also  significantly  lower  in  the  Pathway  condition.  For  the  main 
effect  of  the  visibility  variable,  only  mean  differences  in  rmse  for  airspeed  (VMC  Day  vs.  IMC 
Day)  and  percentage  of  time  off-path  vertically  (VMC  Day  vs.  VMC  Night)  were  significantly 
lower  (in  the  VMC  Day  and  VMC  Night  conditions,  respectively). 
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Figure  22.  Effects  of  primary  flight  display,  visibility,  and  synthetic  terrain  on  Lateral  RMSE. 


For  the  distracter  data,  all  of  the  FTE  means  were  also  significantly  lower  in  the  Pathway 
condition  (see  Figure  23  for  a  representative  graph). 


Figure  23.  Effects  of  primary  flight  display,  visibility,  and  synthetic  terrain  on  Lateral  RMSE  during 

a  secondary  task. 

For  the  visibility  variable,  no  main  effects  were  significant.  However,  for  the  PFD  *  Visibility 
interaction,  a  significant  difference  was  found  for  rmse  vertical  deviation.  This  difference 
occurred  for  both  VMC  Night  and  IMC  Day  vs.  VMC  Day.  In  both  cases,  the  rmse  vertical 
deviation  mean  was  lower  for  VMC  Day  in  the  Mil-Std.  condition  and  higher  for  VMC  Day  in 
the  Pathway  condition. 
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The  performance  of  a  secondary  task  resulted  in  significant  differences  in  the  means  of  absolute 
lateral  deviation,  airspeed  RMSE,  and  all  measures  of  percent  time  off  path.  Further,  the 
interaction  of  PFD  *  Secondary  Task  also  had  a  significant  effect  on  lateral  RMSE  and  all 
percent  off  path  measures.  Effects  on  total  percent  time  off  path  are  shown  in  Figure  24. 
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Figure  24.  Effects  of  primary  flight  display,  visibility,  secondary  task,  and  synthetic  terrain  on 

percent  of  time  off  path. 


Synthetic  terrain  had  no  significant  effect  on  any  of  the  flight  performance  dependent  variables. 

A  secondary  analysis  was  conducted  on  the  FTEs  by  adding  Secondary  Task  to  the  model. 
Unsurprisingly,  results  of  this  model  reveal  main  effects  for  PFD  and  Secondary  Task.  Absolute 
lateral  deviation,  RMS  airspeed  deviation,  and  percent  time  offpath  (total,  lateral,  and  vertical) 
were  all  significant  for  Secondary  Task.  Interestingly,  the  PFD* Secondary  Task  interaction  was 
significant.  Upon  further  examination,  the  means  for  absolute  and  RMS  lateral  deviation  and 
percent  time  offpath  (total,  lateral,  and  vertical)  were  significantly  lower  for  pathway  (see  Figure 
25  for  a  representative  graph). 
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Primary  Flight  Display 

Figure  25.  Effects  of  primary  flight  display,  visibility,  and  synthetic  terrain  on  Lateral  RMSE  during 

a  secondary  task. 


Workload 

A  repeated  measures  multivariate  analysis  of  variance  was  conducted  for  SWORD  and  NASA 
TLX  weighted  workload  levels.  Differences  were  significant  for  both  measures  for  both  primary 
flight  display  and  visibility,  with  the  lower  means  (less  workload)  occurring  in  pathway 
conditions.  For  visibility,  significantly  lower  SWORD  means  occurred  under  VMC  Day 
conditions  vs.  VMC  Night  and  IMC  Day.  The  significantly  lower  TLX  means  only  occurred 
under  VMC  Day  when  contrasted  with  IMC  Day. 

Synthetic  terrain  had  a  significant  effect  on  the  SWORD  variable  only,  with  workload  rated 
lower  when  synthetic  terrain  was  present.  There  was  also  a  significant  effect  on  SWORD  ratings 
of  the  interaction  between  primary  flight  display  and  synthetic  terrain,  with  synthetic  terrain 
rated  as  reducing  workload  only  in  the  MIL-STD  HUD  condition  (see  Figure  26).  Corresponding 
NASA  TLX  data  are  depicted  in  Figure  27  (use  same  legend). 
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Figure  26.  Effects  of  primary  flight  display,  visibility,  and  synthetic  terrain  on  SWORD  ratings. 
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Figure  27.  Effects  of  primary  flight  display,  visibility,  and  synthetic  terrain  on  NASA  TLX  ratings. 

Situation  Awareness 

Included  in  the  MANOVA  above  was  the  SA-SWORD  variable.  For  situation  awareness,  the 
only  main  effeets  signifieant  were  primary  flight  display  and  visibility.  The  SA-SWORD  scores 
had  higher  (more  SA)  means  in  pathway  and  VMC  Day  conditions.  A  significant  interaction  was 
found  between  primary  flight  display  and  synthetie  terrain,  similar  to  that  found  for  SWORD 
data.  Effects  of  independent  variables  on  SA-SWORD  are  shown  in  Figure  28. 
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Figure  28,  Effects  of  primary  flight  display,  visibility,  and  synthetic  terrain  on  SA-SWORD  ratings. 

The  only  significant  effect  on  SAGAT  scores  was  the  effeet  of  visibility  on  overall  SAGAT 
seore.  A  test  of  within-subjeet  contrasts  showed  that  this  effeet  was  due  to  a  difference  between 
VMC  Day  and  the  other  two  visibility  eonditions.  SAGAT  seores  throughout  the  study  were  low 
(pilots  typieally  answered  between  a  third  and  half  of  the  questions  eorreetly)  and  it  is  diffieult  to 
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know  whether  the  lack  of  significant  findings  for  SAGAT  scores  was  due  to  low  statistical  power 
(typical  observed  power  was  0.2),  a  floor  effect,  insensitivity  of  the  measure,  true  invariance  of 
SA  as  independent  variables  were  manipulated,  or  some  combination  of  these.  Even  when  groups 
of  related  questions  were  analyzed  together  (e.g.,  terrain  questions),  results  were  still  not 
significant,  although  trends  were  in  expected  directions.  For  example,  of  responses  received  to 
SAGAT  terrain  questions  across  all  pilots,  57  were  correct  when  synthetic  terrain  was  present, 
versus  40  correct  when  it  was  not.  Similarly,  pilots  answered  3 1  of  these  questions  correctly  in 
the  VMC  Day  condition,  but  only  17  correct  responses  were  received  in  the  IMG  Day  condition 
without  synthetic  terrain. 

Two  other  results  of  note  were  not  analyzed  statistically:  incidence  of  CFIT,  and  response  to 
runway  incursion.  Of  the  260  approaches  flown  during  the  data  collection  phase  of  the  study 
(i.e.,  excluding  practice),  seven  resulted  in  controlled  flight  into  terrain.  All  of  these  occurred  in 
the  IMG  Day  condition  without  synthetic  terrain.  Six  of  the  seven  occurred  when  the  pilot  was 
using  the  MIL-STD  HUD. 

Software/simulation  problems  invalidated  two  of  the  runway  incursion  approaches  (wingtip  and 
tail  lights  on  the  simulated  F-16  failed  to  operate).  Of  the  remaining  eleven  runway  incursion 
approaches,  a  correct  response  to  the  runway  incursion  (initiation  of  a  missed  approach)  was 
observed  on  eight  of  these.  In  the  three  approaches  in  which  a  missed  approach  was  not 
observed,  pilots  executed  a  normal  landing  on  or  very  near  the  simulated  F-16  at  the  touchdown 
point.  Post-experiment  questioning  revealed  that  these  pilots  were  totally  unaware  of  the 
simulated  runway  incursion.  All  of  the  runway  incursion  approaches  in  which  a  missed  approach 
was  not  initiated  occurred  in  the  VMG  Night  condition,  using  the  MIL-STD  HUD,  without 
synthetic  terrain. 

DISCUSSION 

With  respect  to  flight  technical  error,  the  results  of  this  study  replicate  those  of  previous 
comparisons  between  traditional  flight  directors  and  pathway  displays.  Pilots  are  much  better  at 
maintaining  the  commanded  flight  path,  including  airspeed,  when  using  a  pathway-in-the-sky 
display.  Indeed,  pilots  spent  roughly  four  times  as  much  time  outside  the  commanded  corridor 
when  flying  the  MIL-STD  HUD  as  they  did  when  flying  the  Pathway.  The  absolute  magnitude  of 
these  effects  is  very  likely  driven  by  what  was,  in  retrospect,  an  extraordinarily  challenging  task: 
time  spent  learning  the  symbology  sets,  simulator  flight  control  characteristics,  and  studying 
approach  plates  was  much  less  than  what  one  would  expect  of  an  operational  environment. 
Further,  the  paths  flown  represented  something  of  a  “worst  case”  in  terms  of  corridor  dimensions 
and  were  designed  to  test  the  limits  of  pilot  ability  in  a  precision  navigation  environment 
unhindered  by  the  interception  and  tracking  of  radio  navigation  aids.  While  not  a  focus  of  the 
study,  the  results  support  the  current  strategy  of  allowing  pilots  to  fly  a  stabilized  approach  in 
IMG  (one  in  which  requirements  for  control  inputs  are  minimized).  Several  pilots  commented  on 
the  difficulty  imposed  by  the  variety  and  number  of  changes  in  descent  angle  in  the  approaches 
flown. 

Despite  the  demanding  task,  pilots  were  able  to  successfully  fly  the  approaches  when  using  the 
pathway,  even  in  solid  IMG  and  even  in  the  presence  of  distracter  tasks.  The  workload  and  SA 
data  provide  insight  into  why  this  was  the  case.  Better  flight  performance  in  the  pathway 
condition  was  achieved  -  not  at  the  expense  of  increased  workload  -  but  because  use  of  the 
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pathway  reduced  workload.  Indeed,  pilots  commented  that  increased  situation  awareness 
regarding  the  upcoming  path  (and  associated  control  inputs)  allowed  them  to  better  manage 
secondary  tasks.  This  is  evident  in  the  effects  on  flight  performance  of  the  interaction  between 
primary  flight  display  and  secondary  task:  performance  with  the  MIL-STD  HUD  worsened 
significantly  more  with  addition  of  a  secondary  task  than  did  performance  with  the  pathway.  The 
NASA-TLX  data  show  that  workload  decreased  roughly  20%  with  use  of  the  pathway,  a 
decrease  that  may  be  practically  significant.  Experience  with  this  metric  indicates  that  a 
“redline”,  a  point  at  which  performance  begins  to  significantly  deteriorate,  may  be  around  50 
(e.g.,  Aretz,  Johannsen,  &  Ober,  1995).  In  the  current  study,  pilots  rated  their  workload  near  this 
value  when  using  the  pathway,  but  well  beyond  it  when  using  the  MIL-STD  HUD. 

While  the  results  with  respect  to  primary  flight  display  are  not  unexpected,  the  results  with 
respect  to  synthetic  terrain  do  contain  some  surprises.  Synthetic  terrain  appeared  to  have  no 
effect  on  flight  performance  and  affected  only,  1)  the  most  sensitive  measures  of  workload  and 
SA  (SWORD  and  SA-SWORD),  and,  2)  only  in  the  MIL-STD  HUD  condition.  As  described 
previously,  analyses  of  SAGAT  results  were  problematic,  but  these  data  did  trend  toward  an 
increase  in  terrain  S  A  with  the  inclusion  of  synthetic  terrain.  While  caution  should  be  exercised 
in  basing  any  conclusions  on  only  seven  events,  the  fact  that  no  CFITs  occurred  when  synthetic 
terrain  was  present  seems  promising. 

The  subjective  questionnaire  asked  pilots  to  rate  the  usefulness  of  the  MIL-STD  HUD,  the 
pathway,  and  synthetic  terrain  on  a  scale  from  -3  to  +3,  with  -3  being  “Extremely  Low”  and  +3 
being  “Extremely  High”.  The  average  ratings  were  -1.2,  2.4,  and  1.2,  respectively.  A  majority  of 
pilots  commented  that  the  synthetic  terrain  would  be  most  useful  in  IMC  or  at  night.  Five  of  the 
thirteen  commented  that  they  would  want  control  of  the  brightness  or  contrast  for  the  synthetic 
terrain  (to  include  decluttering  it  entirely),  especially  on  short  final.  Several  of  the  pilots  would 
have  decluttered  the  follow-me  aircraft  if  they  had  had  the  option,  and  there  were  several 
suggestions  for  adding  reference  markers  (e.g.,  airspeed,  altitude,  and  heading  “bugs”)  to  the 
pathway  display  to  support  awareness  of  basic  approach  parameters,  especially  in  the  event  of 
display  failure. 

While  pilots’  ratings  of  pathway  utility  were  quite  high,  two  of  the  thirteen  made  comments  that 
indicated  a  concern  relative  to  non-pathway  SA: 

Pilot  A:  “Very  useful  in  helping  the  pilot  predict  where  the  flight  path  of  the  aircraft  would  be 
relative  to  current  ownship  attitude  and  heading.  However,  SA  on  actual  path  segment  headings 
or  commanded  altitudes  was  not  high.  Could  get  complacent  "following  the  path",  leading  to 
pilots  not  maintaining  overall  orientation  to  the  approach.” 

Pilot  B:  “The  pathway  format  reduced  my  workload  greatly  once  I  acquired  a  better 
understanding  of  the  system.  Easy  to  interpret  the  displayed  information.  The  danger  I  see  here  is 
a  pilot  can  very  easily  tune  out  the  world  around  him  while  totally  focusing  on  the  pathway.” 

While  these  comments  would  support  a  hypothesis  of  cognitive  capture  (or  attentional  tunneling) 
associated  with  the  pathway,  they  are  somewhat  belied  by  the  runway  incursion  results:  none  of 
the  incorrect  responses  to  the  runway  incursion  occurred  in  the  pathway  condition.  Rather,  the 
results  support  an  alternate  hypothesis  that  -  even  for  an  unexpected  event  -  the  conformal 
nature  of  pathway  HUD  symbology  (especially  in  combination  with  a  synthetic  runway  outline) 
and  its  head-up  location  facilitate  SA,  at  least  relative  to  events  in  the  far  domain  near  to  or 
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overlaid  by  the  symbology  (Fadden,  Ververs,  &  Wickens,  2001;  Martin-Emerson  &  Wickens, 
1997). 


Conclusions 

As  applied  research  comes  closer  to  actual  application,  care  must  be  taken  in  overgeneralizing 
results.  Among  things  to  bear  in  mind  in  interpreting  the  results  of  this  study  and  related  studies 
that  have  been  published  in  recent  years  is  that  pilots  were  flying  a  part-task  simulation.  Pilot 
performance,  workload,  and  SA  are  greatly  influenced  by  several  factors  common  in  actual 
aviation  environments  that  are  not  common  in  part-task  simulations.  Important  among  these  are 
communications  with  air  traffic  management,  the  presence  of  other  crewmembers,  and  physical 
stresses  (e.g.,  G,  cold/heat,  turbulence). 

However,  with  respect  to  flight  technical  error,  the  results  of  this  study  replicate  those  of 
previous  comparisons  between  traditional  flight  directors  and  pathway  displays  both  in 
simulation  and  in  flight  (Theunissen,  1997;  Fadden,  Ververs,  &  Wickens,  2001).  In  comparison 
to  standard  2D  flight  directors,  pilots  are  much  better  at  maintaining  a  commanded  path  when 
using  a  pathway  display  and  it  seems  likely  that  this  effect  is  magnified  with  increasing  path 
complexity.  When  commanded  corridor  dimensions  (and  associated  C/R  ratios)  are  held 
constant,  workload  is  reduced  by  use  of  a  pathway  display  and  situation  awareness  is  increased. 
The  increase  in  awareness  of  future  path-related  events,  especially  those  associated  with  control 
inputs,  reduces  pilot  workload  and  allows  better  management  of  secondary  tasks. 

The  current  study  certainly  does  not  rule  out  the  phenomenon  of  cognitive  capture  or  attentional 
tunneling  associated  with  pathway  displays.  However,  it  does  support  a  hypothesis  that  any  such 
detriment  can  be  alleviated  via  the  placement  of  conformal  symbology  in  a  head-up  location. 
Extrapolating,  concerns  about  decreased  awareness  of  air  traffic  associated  with  pathway  usage 
should  not  be  addressed  by  using  symbology  other  than  a  pathway,  but  by  including  conformal 
overlays  for  traffic  in  the  HUD.  In  pursuing  this  strategy,  research  is  needed  concerning  trade¬ 
offs  between  conformal  overlays  and  amount  of  clutter.  Given  all  the  potentially  useful  database- 
based  information  that  could  be  displayed  to  a  pilot  (e.g.,  terrain,  traffic,  flight  path,  atmospheric 
phenomena,  airspace  boundaries),  there  is  the  potential  to  render  a  HUD  informationally  opaque 
and  not  useful  for  its  original  intended  purpose. 
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VI.  MISSION  FACTORS  IMPACTING  SYNTHETIC  VISION  AND  FUTURE  COCKPIT 

CONCEPTS 


CREW  TASKS  AND  AIRCRAFT  MISSION 

Specific  tasks  or  missions  require  different  synthetic  vision  formats.  Just  like  the  civil  world,  the 
military  owns  different  types  of  aircraft  to  perform  different  types  of  missions.  The  C-17  can 
perform  tactical  airlift,  the  F- 16  and  F/A-18  provide  air  interdiction,  close  air  support,  and 
offensive/defensive  counterair,  while  the  E-3  Airborne  Warning  and  Control  System  (AWACS) 
manages  the  command  and  control  aspects  for  other  joint  platforms  in-theater.  Such  wide- 
ranging  missions  require  different  platforms  and  subsystems,  and  synthetic  vision  systems  are 
among  these. 

The  E-3,  whose  mission  it  is  to  fly  high  enough  to  maintain  command  authority  over  a  certain 
region  of  airspace  only  comes  near  the  ground  during  takeoff  and  landing  operations.  Synthetic 
vision  systems  requirements  for  this  type  of  mission  are  similar  to  those  of  the  civil  world, 
including  takeoff/landing  in  fog.  Precise  geopositioning  capabilities  through  GPS/DGPS 
technology  are  necessary  for  terminal  operations,  but  little  else. 

On  the  other  hand,  an  aircraft  engaged  in  close  air  support  in  an  area  of  varied  terrain  may  be 
flying  near  the  ground  with  very  little  preplanning  due  to  the  dynamic  nature  of  engaging 
camouflaged  and  concealed  enemy  ground  forces.  Complicating  this  task  even  further  is  the 
possible  proximity  of  friendly  ground  forces,  increasing  the  risk  of  fratricide.  Throw  instrument 
meteorological  conditions  into  the  mix  and  the  pilot’s  task  grows  more  and  more  difficult.  A 
synthetic  view  of  the  local  area  could  conceivably  improve  a  pilot’s  awareness  of  surrounding 
terrain  to  that  near  the  level  of  a  clear  and  sunny  day.  Indeed,  synthetic  vision  systems  have  the 
potential  to  provide  more  and  better  depth  and  self-motion  cues  than  are  normally  available  to  a 
pilot  on  such  a  clear  and  sunny  day.  Additionally,  because  the  imagery  can  be  at  least  partly 
database  driven,  the  database  could  contain  symbology  for  ground  forces,  both  friendly  and 
enemy,  to  improve  lethality  and  reduce  the  probability  of  friendly  fire  casualties,  assuming 
accurate  intelligence  can  be  fed  into  the  database  in  near  real  time.  The  capability  to  update  the 
onboard  database  through  a  datalink  with  offboard  sources  (such  as  the  AWACS,  Joint 
Surveillance  Target  Attack  Radar  System  (JSTARS),  and  Forward  Air  Controller  (FAC),  etc.) 
after  arriving  in  the  Area  of  Operations  increases  the  chance  of  a  successful  mission. 

LASER  THREAT  PROTECTION  AND  NIGHT/WEATHER  VISION 

Another  consideration  for  the  use  of  SVS  is  the  increasing  use  of  battlefield  directed  energy  such 
as  lasers.  Possible  enemy  use  of  lasers  to  blind  friendly  forces  is  leading  to  the  development  of 
laser  eye  protection  measures,  at  this  point  in  time  consisting  primarily  of  static  vision  filters  for 
the  common  wavelengths  of  lasers  in  use.  Of  course,  the  use  of  a  filter  for  a  visible  laser  is  going 
to  obstruct  the  vision  of  the  pilot  to  some  degree.  Depending  on  where  the  filter  is  placed,  either 
included  in  the  canopy  or  special  glasses  on  the  pilot,  the  use  of  an  SVS  could  compensate  for 
the  frequency-specific  “notches”  in  the  pilot’s  vision  by  supplementing  the  real  out  the  window 
scene.  Unless  some  non-vision-obscuring  method  for  protecting  aircrew  from  blinding  energy  is 
discovered,  the  view  “out  the  window”  may  become  more  and  more  occluded  and  SVS  may  be 
called  upon  to  make  up  the  difference. 

The  direct  view  now  acquired  by  the  pilot's  unaided  eyes  looking  out  of  current  cockpits  might 
be  denied  even  during  a  clear  weather  day  by  directed  energy  threats.  Rules  of  engagement. 
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however,  require  human-in-the-loop  to  the  last  moment  possible  before  munitions  release  and 
during  fly-out  to  ground  targets  to  minimize  collateral  damage  and  civilian  casualties.  In 
addition,  combat  pilots  suffer  from  information  overload  resulting  in  loss  of  situational 
awareness  at  times  when  it  is  most  important:  beyond  visual  range  objects  are  difficult  to 
envision  and  fit  into  a  total  picture;  sensor  advancements  provide  ever  higher  resolution  targeting 
imagery  real  time  in  the  cockpit.  These  threats,  together  with  night,  in- weather,  low-level  flight 
conditions  are  giving  rise  to  large  head-down  panoramic  displays  in  2 1  st  century  cockpits  and 
make  the  case  for  exploration  of  immersive  displays. 

As  lasers  become  more  ubiquitous,  the  future  military  pilot — and  even  civilian — might  have  to 
fly  some  flight  segments  without  looking  out  of  the  cockpit.  The  canopy  would  be  closed  by 
curtains  or  by  an  electrically  controlled  opaquing  layer — only  during  these  times.  Then  a 
synthetically  generated  view  of  the  real  world  would  be  created.  The  control  and  display  system 
might  logically  evolve  as  an  extension  of  present  day  night/in-weather  instrumented  flight 
systems.  A  more  extensive  in-cockpit  display  suite  may  become  necessary  to  survival  and 
mission  success.  Such  a  system  might  include  a  4000  cm^  (600  in^)  head-down  color  MFD  that 
would  be  viewable  in  sunlight  or  starlight,  plus  a  helmet-mounted  cueing  system.  The  opaqued 
canopy  might  include  a  closable  display  viewable  just  in  dim  ambient  to  provide  simulated 
vision  over  the  full  field  of  view  denied  episodically  by  external  conditions.  Ideally  these 
displays  would  be  physically  redundant  yet  appear  seamless. 

PANORAMIC  COCKPIT  CONCEPT 

A  program  of  studies  conducted  by  the  Air  Force  Research  Laboratory  in  1990  demonstrated  an 
effective  way  to  begin  to  deal  with  the  threat  and  synthetic  vision  issues  just  described.  The 
approach  in  this  program,  entitled  "Panoramic  Cockpit  Control  and  Display  System  (PCCADS)," 
was  to  provide  the  pilot  with  large  area  displays  and  a  helmet-mounted  off-axis  target-acquisition 
weapon-targeting  system.  There  were  two  projects,  one  focused  near  term,  one  far  term  (Hopper, 
1992). 

The  PCCADS  2000  cockpit  was  designed  to  be  realizable  with  1995  technology  with  production 
by  2000  and  featured  a  25  cm  (10  in.)  square  tactical  situation  display  and  two  15  cm  (6  in.) 
square  secondary  multifunction  displays  on  either  side.  All  displays  were  full  color  capable  with 
a  total  area  of  1 1 10  cm^  (172  in^).  The  test  mission  was  for  an  F-15E.  A  28%  increase  in 
exchange  ratio  was  achieved  versus  the  standard  F-15E  cockpit.  An  18%  increase  was  observed 
for  the  addition  of  helmet  cueing  to  the  F-15E  baseline  cockpit.  Coupling  this  large  display  with 
a  helmet-mounted  cueing  system  (HMCS)  for  off  axis  target  acquisition  resulted  in  a  45% 
increase.  The  F-22A  Raptor  will  realize  the  PCCADS  2000  concept  in  a  production  cockpit 
(video  wall  comprising  six  separate  flat  panel  AMLCDs  with  having  an  aggregate  resolution  of 
1.35  megapixels  at  5-bit  grayscale  (32  gray  levels)  per  color  in  1290  cm^  (201  in^)  plus  Joint 
Helmet  Mounted  Cueing  System  (JHMCS)  add-on.  Beyond  the  PCCADS  2000  cockpit  was  full 
PCCADS  with  a  continuous  display  surface  of  200-300  in^. 

SUPER  PANORAMIC  COCKPIT  (SPC)  CONCEPT 

Hopper  (2000b,  2000c)  introduced  a  super-panoramic  cockpit  (SPC)  concept  to  advance 
discussion  of  features  that  might  be  explored  to  implement  synthetic  vision  in  both  open  and 
closed  cockpits.  The  SPC  concept  is  illustrated  in  Figure  11.  The  SPC  concept  has  a  central  large 
area  display  a  la  PCCADS  surrounded  left  and  right  by  similarly  large,  curved  cockpit- 
conformable  displays  with  the  goal  of  creating  a  HDD  system  with  over  100°  FOV.  The  100° 
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FOV  is  an  important  metric  for  significantly  greater  engagement  of  the  capabilities  of  the  human 
vision  system.  The  HUD  is  supplemented  by  a  set  of  deployable  flat  panel  displays  to  create  a 
second  100“  FOV  in  the  direction  of  forward  view;  this  system  affords  night/in- weather/in-laser 
flying  capability  upon  demand.  In  addition  ,  the  SPC  concept  includes  a  closable  curtain  inside 
of  the  canopy  in  the  near  term  that  gives  way  to  a  flexible  canopy  display  system  (CDS)  in  the 
far  term.  Deployment  of  both  the  HUD  panoramic  FPD  HUD  system  and  the  CDS  would  be 
mechanized  and  typically  actuated  only  upon  pilot  command.  The  deployable  FPDs  and  CDS 
may  initially  be  build  using  current  display  technology.  In  the  far  term  the  FPDs  and  CDS  may 
become  roll-up  flexible  displays  based  on  technology  now  being  created  by  the  Flexible  Display 
Program  at  the  Defense  Advanced  Research  Projects  Agency. 

Beyond  the  SPC  concept  is  the  immersive  cockpit,  which  provides  a  maximum  possible,  4n 
steradian,  FOV  at  the  resolution  of  the  natural  world.  For  2-D  hardware  this  system  would 
require  some  1  billion  pixels  to  provide  for  just  20/20  visual  acuity. 


SUPERPANORAMIC  COCKPIT  WITH  CLOSABLE  OPAQUE  LAYER 


HUD  Screen  Support: 
Attenuates  Bird  Strike; 
Braces  Inner  Layer 


CLOSABLE  INNER  LAYER  (CIL): 

Near  term:  SHAPED  CURTAIN  ON  RAILS 
Mid  term:  FLEXIBLE  DISPLAYS  IN  A  SHELL 
Far  term:  CANOPY  DISPLAY  SYSTEM  (CDS) 


CANOPY 


Layer 

Stowage 


OPEN 

DISPLAY  SYSTEM: 
Panoramic  Center  Screen 
+  Between  Knees 
+  Left/Right  Curved  Screens 
+  Multifunction  HUD 


CLOSED 

DISPLAY  SYSTEM: 

Open  Display  System  +  CDS 
(CIL  is  segmented  w/FPDs  or 
is  continuous  flexible  display) 
+  Acoustic  +  Haptic  (touch) 


Figure  29.  Super  panoramic  cockpit  with  deployable  flat  panel  forward  vision  system  and  flexible 

canopy  display  system. 
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DISCUSSION 

Fielded  military  cockpits  have  crossed  the  megapixel  threshold  (F-22A,  AWACS/JSTARS 
crewstation).  The  human  visual  system  is  capable  of  digesting  one  gigapixel,  color  high 
grayscale  full  motion  video  in  2D  and  even  more  if  3D  is  considered.  A  substantial  closing  of  this 
human  vs.  fielded  gap  will  make  new  systems  affordable  by  reducing  both  crew  sizes  and 
training/rehearsal  flights  while  increasing  individual  effectiveness.  A  corollary  will  be  to  provide 
the  means  to  operate  more  effectively  during  attack  mission  segments  in  which  direct  vision 
though  a  canopy  is  denied  by  night,  weather,  or  anti-personnel  directed  energy.  Advances  in 
display  technology  and  the  advent  of  digital  television  will  enable  a  100-fold  growth  in  cockpit 
resolution  by  2010.  Simulators  and  trainer  systems  might  leverage  this  technology  trend  to 
produce  tactical  fighter  aerospace  canopy  vision  systems  at  20-20  resolution  (170  megapixel 
versus  present  16  megapixel).  Such  20-20  simulators  could  save  jet  fuel  by  reducing  training 
flights,  increase  effectiveness  of  pre-mission  rehearsal,  and  enable  realistic  human  factors 
research  on  directed  energy  threats.  Uninhabited  combat  vehicles  (UCV)  may  require  addition  of 
inset  displays  to  present  high  resolution  sensors:  video  (FLIR,  LLLTV  at  25  megapixel  per 
frame)  and  stills  (4  megapixel  satellite  photographs  and  significant  portions  of  8-30  gigapixel 
battlefield  terrain  databases).  Work  on  advanced  simulators  and  UCVs,  as  well  as  C41  data  walls 
and  complete  audio-visual  environments  (CAVEs),  prepare  the  way  for  in-aircraft  in-spacecraft 
hectomegapixel  display  systems  with  200  megapixels  (160  immersive  canopy,  35  panoramic 
head-down,  5  see-through  head-mounted).  The  hardware  technology  is  being  developed  to 
creation  of  panoramic  and  immersive  cockpit  display  systems  to  implement  synthetic  vision. 

We  note  that  the  largest  display  in  the  F-22A  cockpit  is  just  7.8  x  7.8  in.  compared  to  the  15  x  15 
in.  minimum  size  one  would  need  to  present  imagery  from  current  sensors  to  the  pilot  without 
distortion  or  cropping.  And  even  the  F-22A  is  not  yet  fielded.  Thus,  the  display  size  needs  to 
grow  to  support  both  the  full  use  of  current  sensors  as  well  as  the  advanced  sensors  which 
already  have  demonstrated  even  larger  fields  of  view  and  resolutions  over  25  megapixels  per 
frame.  Cockpit  retrofits  as  well  as  new  systems  programs  are  envisioned  for  implementation. 

Rules  of  engagement  and  technology  limits  (transmission  bandwidth,  speed)  argue  for  the 
retention  of  a  pilot  in  aerospace  craft,  albeit  with  an  immersive  display  capability  activated  for 
attack  mission  segments,  in  many  air  and  space  combat  platforms. 

An  igloo  made  of  eight  25  megapixel  AMLCD  flat  panel  displays  may  be  a  near  term  incarnation 
of  the  hectomegapixel  cockpit  display.  In  the  far  term,  flexible  displays  on  large  thin  substrates 
(steel,  plastic),  produced  with  printing  technology,  may  permit  the  design  of  a  closable  inner 
cockpit  shell  with  160  megapixels  on  the  inside  and  a  deflecting/absorbing  layer  on  the  outside. 

Eventually,  the  aggregate  resolution  of  an  immersive  cockpit  might  need  to  be  250  megapixels,  a 
visual  photon  information  flow  to  the  pilot  of  360  gigabits  per  second. 
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